I am currently developing a module for OpenERP 6.1.1 (python 2.7) where email notifications shall be triggered by several workflow state changes. So far, so obvious. When I configure an instance of ir.mail_server and want to test it, I get the following exception:
Server Traceback (most recent call last):
File "/opt/openerp61/server/openerp/addons/web/common/http.py", line 592, in send
result = openerp.netsvc.dispatch_rpc(service_name, method, args)
File "/opt/openerp61/server/openerp/netsvc.py", line 360, in dispatch_rpc
result = ExportService.getService(service_name).dispatch(method, params)
File "/opt/openerp61/server/openerp/service/web_services.py", line 572, in dispatch
res = fn(db, uid, *params)
File "/opt/openerp61/server/openerp/osv/osv.py", line 167, in execute_kw
return self.execute(db, uid, obj, method, *args, **kw or {})
File "/opt/openerp61/server/openerp/osv/osv.py", line 121, in wrapper
return f(self, dbname, *args, **kwargs)
File "/opt/openerp61/server/openerp/osv/osv.py", line 176, in execute
res = self.execute_cr(cr, uid, obj, method, *args, **kw)
File "/opt/openerp61/server/openerp/osv/osv.py", line 164, in execute_cr
return getattr(object, method)(cr, uid, *args, **kw)
File "/opt/openerp61/server/openerp/addons/base/ir/ir_mail_server.py", line 191, in test_smtp_connection
smtp_debug=smtp_server.smtp_debug)
File "/opt/openerp61/server/openerp/addons/base/ir/ir_mail_server.py", line 241, in connect
connection.login(user, password)
File "/usr/lib/python2.7/smtplib.py", line 598, in login
(code, resp) = self.docmd(encode_cram_md5(resp, user, password))
File "/usr/lib/python2.7/smtplib.py", line 562, in encode_cram_md5
response = user + " " + hmac.HMAC(password, challenge).hexdigest()
File "/usr/lib/python2.7/hmac.py", line 72, in __init__
self.outer.update(key.translate(trans_5C))
TypeError: character mapping must return integer, None or unicode
The problem seems obvious, as user and password arguments passed to smtplib.SMTP.login() are unicode encoded, which HMAC doesn't like. If I "fix" the OpenERP core and cast these arguments to string everything seems to work fine. At least the "Test Connection" functionality of ir.mail_server says so.
As I am learning OpenERP as well as Python with this project, I don't know how to proceed though, since there are almost no references to anyone else having this problem. Therefore, my guess is that there is something "wrong" with my development setup triggering this problem. I could just leave the core-patch in there and continue development but this is not really an option, as this might just come back one day and bite my ass.
Any input on this would be great.
You are right, HMAC doesn't like unicode. You can apply the following fix, provided you only use ASCII chars in usernames and passwords. (you have to change line 241 of /opt/openerp61/server/openerp/addons/base/ir/ir_mail_server.py yourself to apply this)
connection.login(str(user), str(password))
There's a bug in Trac regarding this:
http://bugs.python.org/issue5285
Related
i got work to fix error when computing but i still dont have idea how to fix it because i'm still newbie
Odoo Server Error
Traceback (most recent call last): File
"/home/equipAccounting/equip/odoo/addons/base/models/ir_http.py", line
237, in _dispatch
result = request.dispatch() File "/home/equipAccounting/equip/odoo/http.py", line 683, in dispatch
result = self._call_function(**self.params) File "/home/equipAccounting/equip/odoo/http.py", line 359, in
_call_function
return checked_call(self.db, args, *kwargs) File "/home/equipAccounting/equip/odoo/service/model.py", line 94, in
wrapper
return f(dbname, args, *kwargs) File "/home/equipAccounting/equip/odoo/http.py", line 347, in checked_call
result = self.endpoint(*a, **kw) File "/home/equipAccounting/equip/odoo/http.py", line 912, in call
return self.method(*args, **kw) File "/home/equipAccounting/equip/odoo/http.py", line 531, in response_wrap
response = f(*args, **kw) File "/home/equipAccounting/equip/addons/basic/web/controllers/main.py",
line 1393, in call_button
action = self._call_kw(model, method, args, kwargs) File "/home/equipAccounting/equip/addons/basic/web/controllers/main.py",
line 1381, in _call_kw
return call_kw(request.env[model], method, args, kwargs) File "/home/equipAccounting/equip/odoo/api.py", line 396, in call_kw
result = _call_kw_multi(method, model, args, kwargs) File "/home/equipAccounting/equip/odoo/api.py", line 383, in _call_kw_multi
result = method(recs, args, *kwargs) File "/home/equipAccounting/equip/addons/core/treasury_forecast/models/treasury_bank_forecast.py",
line 290, in compute_bank_balances
self.env.cr.execute(main_query) File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in
fun
return caller(func, (extras + args), *kw) File "/home/equipAccounting/equip/odoo/sql_db.py", line 101, in check
return f(self, args, *kwargs) File "/home/equipAccounting/equip/odoo/sql_db.py", line 298, in execute
res = self._obj.execute(query, params) Exception
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File
"/home/equipAccounting/equip/odoo/http.py", line 639, in
_handle_exception
return super(JsonRequest, self)._handle_exception(exception) File "/home/equipAccounting/equip/odoo/http.py", line 315, in
_handle_exception
raise exception.with_traceback(None) from new_cause psycopg2.errors.SyntaxError: syntax error at or near ")" LINE 9:
WHERE abs.journal_id IN ()
and here is the code :
def get_bank_fc_query(self, fc_journal_list, date_start, date_end,company_domain):
query = """
UNION
SELECT CAST('FBK' AS text) AS type, absl.id AS ID, am.date, absl.payment_ref as name, am.company_id, absl.amount_main_currency as amount, absl.cf_forecast, abs.journal_id, NULL as kind FROM account_bank_statement_line absl
LEFT JOIN account_move am ON (absl.move_id = am.id)
LEFT JOIN account_bank_statement abs ON (absl.statement_id = abs.id)
WHERE abs.journal_id IN {}
AND am.date BETWEEN '{}' AND '{}'
AND am.company_id in {} """
.format(str(fc_journal_list), date_start, date_end,company_domain)
return query
def get_acc_move_query(self, date_start, date_end, company_domain):
query = """
UNION
SELECT CAST('FPL' AS text) AS type, aml.id AS ID,aml.treasury_date AS date, am.name AS name, aml.company_id, aml.amount_residual AS amount, NULL AS cf_forecast,
NULL AS journal_id, am.move_type as kind
FROM account_move_line aml
LEFT JOIN account_move am ON (aml.move_id = am.id)
WHERE am.state NOT IN ('draft')
AND aml.treasury_planning AND aml.amount_residual != 0
AND aml.treasury_date BETWEEN '{}' AND '{}'
AND aml.company_id in {} """
.format(date_start, date_end, company_domain)
return query
Thanks in advance
The error has nothing to do with Odoo.
psycopg2.errors.SyntaxError: syntax error at or near ")" LINE 9:
WHERE abs.journal_id IN ()
It's cleary a syntax error in the query itself. You're using the IN operator without having a value list afterwards.
Your fc_journal_list parameter doesn't have values on your call. You should catch an empty list before creating the query.
And then there are atleast 2 big security risks in your code:
never ever use string formatting for querys, the comment under your question already points to variables in SQL queries that's the common mistake to make SQL injections an easy thing...
don't make such security risky methods (here both query returning methods) public to the odoo external API. Just add a _ at the beginning of the method names and you're fine on that part.
Odoo has a very powerful ORM API to do the psql queries. Is there a good reason you use sql instead?
The functions you need are, Read for selecting the fields you use, search and filtered for filtering the results.
I suggest reading the following tutorial.
https://www.odoo.com/documentation/14.0/developer/reference/addons/orm.html#search-read
also look at good examples inside the odoo source, I think the stock module is a good place to see some examples.
https://github.com/odoo/odoo/blob/14.0/addons/stock/models/stock_move.py
To fix the error without removing the query, In the function calling get_bank_fc_query you have to check for empty lists first. In python that is very easy, becouse everything that is empty equals False, so do this:
if not fc_journal_list:
raise exceptions.UserError('fc_journal_list cannot be empty')
query = self.get_bank_fc_query(fc_journal_list, date_start, date_end,company_domain
....
I'm getting this error of autoreconnect, and there are around 100 connections in the logs during this call to .objects. Here is the document:
class NotificationDoc(Document):
patient_id = StringField(max_length=32)
type = StringField(max_length=32)
sms_sent = BooleanField(default=False)
email_sent = BooleanField(default=False)
sending_time = DateTimeField(default=datetime.utcnow)
def __unicode__(self):
return "{}_{}_{}".format(self.patient_id, self.type, self.sending_time.strftime('%Y-%m-%d'))
and this is the query call that causes the issue:
docs = NotificationDoc.objects(sending_time__lte=from_date)
This is the complete stacktrace. How can I understand what is going on?
pymongo.errors.AutoReconnect: connection pool paused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
return self.run(*args, **kwargs)
File "/app/src/reminder/configurations/app/worker.py", line 106, in schedule_recall
schedule_recall_use_case.execute()
File "/app/src/reminder/domain/notification/recall/use_cases/schedule_recall.py", line 24, in execute
notifications_sent = self.notifications_provider.find_recalled_notifications_for_date(no_recalls_before_date)
File "/app/src/reminder/data_providers/database/odm/repositories.py", line 42, in find_recalled_notifications_for_date
if NotificationDoc.objects(sending_time__lte=from_date).count() > 0:
File "/usr/local/lib/python3.8/site-packages/mongoengine/queryset/queryset.py", line 144, in count
return super().count(with_limit_and_skip)
File "/usr/local/lib/python3.8/site-packages/mongoengine/queryset/base.py", line 423, in count
count = count_documents(
File "/usr/local/lib/python3.8/site-packages/mongoengine/pymongo_support.py", line 38, in count_documents
return collection.count_documents(filter=filter, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pymongo/collection.py", line 1502, in count_documents
return self.__database.client._retryable_read(
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1307, in _retryable_read
with self._secondaryok_for_server(read_pref, server, session) as (
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1162, in _secondaryok_for_server
with self._get_socket(server, session) as sock_info:
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1099, in _get_socket
with server.get_socket(
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 1371, in get_socket
sock_info = self._get_socket(all_credentials)
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 1436, in _get_socket
self._raise_if_not_ready(emit_event=True)
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 1407, in _raise_if_not_ready
_raise_connection_failure(
File "/usr/local/lib/python3.8/site-packages/pymongo/pool.py", line 250, in _raise_connection_failure
raise AutoReconnect(msg) from error
pymongo.errors.AutoReconnect: mongo:27017: connection pool paused
turned out to be celery is leaving connections open, so here is the solution
from mongoengine import disconnect
from celery.signals import task_prerun
#task_prerun.connect
def on_task_init(*args, **kwargs):
disconnect(alias='default')
connect(db, host=host, port=port, maxPoolSize=400, minPoolSize=200, alias='default')
This could be related to the fact that PyMongo is not fork-safe. If you're using a process pool in any way, which includes server software like uWSGI and even some configurations of popular ASGI servers.
Every time you run a query in PyMongo, that MongoClient object becomes fork-unsafe. A MongoClient object that has never run any queries is fork-safe. Created Database and Collection objects will reference back to their parent MongoClient without checking for existence.
I do see you are using MongoEngine, which uses PyMongo underneath. I don't know the semantics of that particular library but I would assume they are the same.
In short: you must re-create your connections as part of your forking process.
pip install -U pymongo; is already fix; see this https://github.com/mongodb/mongo-python-driver/pull/944
I'm having trouble identifying the source of transaction.interfaces.NoTransaction errors within my Pyramid App. I don't see any patterns to when the error happens, so to me it's quite random.
This app is a (semi-) RESTful API and uses SQLAlchemy and MySQL. I'm currently running within a docker container that connects to an external (bare metal) MySQL instance on the same host OS.
Here's the stack trace for a login attempt within the App. This error happened right after another login attempt that was actually successful.
2020-06-15 03:57:18,982 DEBUG [txn.140501728405248:108][waitress-1] new transaction
2020-06-15 03:57:18,984 INFO [sqlalchemy.engine.base.Engine:730][waitress-1] BEGIN (implicit)
2020-06-15 03:57:18,984 DEBUG [txn.140501728405248:576][waitress-1] abort
2020-06-15 03:57:18,985 ERROR [waitress:357][waitress-1] Exception while serving /auth
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/waitress/channel.py", line 350, in service
task.service()
File "/usr/local/lib/python3.8/site-packages/waitress/task.py", line 171, in service
self.execute()
File "/usr/local/lib/python3.8/site-packages/waitress/task.py", line 441, in execute
app_iter = self.channel.server.application(environ, start_response)
File "/usr/local/lib/python3.8/site-packages/pyramid/router.py", line 270, in __call__
response = self.execution_policy(environ, self)
File "/usr/local/lib/python3.8/site-packages/pyramid_retry/__init__.py", line 127, in retry_policy
response = router.invoke_request(request)
File "/usr/local/lib/python3.8/site-packages/pyramid/router.py", line 249, in invoke_request
response = handle_request(request)
File "/usr/local/lib/python3.8/site-packages/pyramid_tm/__init__.py", line 178, in tm_tween
reraise(*exc_info)
File "/usr/local/lib/python3.8/site-packages/pyramid_tm/compat.py", line 36, in reraise
raise value
File "/usr/local/lib/python3.8/site-packages/pyramid_tm/__init__.py", line 135, in tm_tween
userid = request.authenticated_userid
File "/usr/local/lib/python3.8/site-packages/pyramid/security.py", line 381, in authenticated_userid
return policy.authenticated_userid(self)
File "/opt/REDACTED-api/REDACTED_api/auth/policy.py", line 208, in authenticated_userid
result = self._authenticate(request)
File "/opt/REDACTED-api/REDACTED_api/auth/policy.py", line 199, in _authenticate
session = self._get_session_from_token(token)
File "/opt/REDACTED-api/REDACTED_api/auth/policy.py", line 320, in _get_session_from_token
session = service.get(session_id)
File "/opt/REDACTED-api/REDACTED_api/service/__init__.py", line 122, in get
entity = self.queryset.filter(self.Meta.model.id == entity_id).first()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3375, in first
ret = list(self[0:1])
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3149, in __getitem__
return list(res)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3502, in _execute_and_instances
conn = self._get_bind_args(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3517, in _get_bind_args
return fn(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3496, in _connection_from_session
conn = self.session.connection(**kw)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1138, in connection
return self._connection_for_bind(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1146, in _connection_for_bind
return self.transaction._connection_for_bind(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 458, in _connection_for_bind
self.session.dispatch.after_begin(self.session, self, conn)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/event/attr.py", line 322, in __call__
fn(*args, **kw)
File "/usr/local/lib/python3.8/site-packages/zope/sqlalchemy/datamanager.py", line 268, in after_begin
join_transaction(
File "/usr/local/lib/python3.8/site-packages/zope/sqlalchemy/datamanager.py", line 233, in join_transaction
DataManager(
File "/usr/local/lib/python3.8/site-packages/zope/sqlalchemy/datamanager.py", line 89, in __init__
transaction_manager.get().join(self)
File "/usr/local/lib/python3.8/site-packages/transaction/_manager.py", line 91, in get
raise NoTransaction()
transaction.interfaces.NoTransaction
The trace shows that the execution eventually reaches my project, but only my custom authentication policy. And it fails right where the database should be queried for the user.
What intrigues me here is the third line on the stack trace. It seems Waitress somehow aborted the transaction it created? Any clue why?
EDIT: Here's the code where that happens: policy.py:320
def _get_session_from_token(self, token) -> UserSession:
try:
session_id, session_secret = self.parse_token(token)
except InvalidToken as e:
raise SessionNotFound(e)
service = AuthService(self.dbsession, None)
try:
session = service.get(session_id) # <---- Service Class called here
except NoResultsFound:
raise SessionNotFound("Invalid session found Request headers. "
"Session id: %s".format(session_id))
if not service.check_session(session, session_secret):
raise SessionNotFound("Session signature does not match")
now = datetime.now(tz=pytz.UTC)
if session.validity < now:
raise SessionNotFound(
"Current session ID {session_id} is expired".format(
session_id=session.id
)
)
return session
And here is an a view on the that service class method:
class AuthService(ModelService):
class Meta:
model = UserSession
queryset = Query(UserSession)
search_fields = []
order_fields = [UserSession.created_at.desc()]
# These below are from the generic ModelClass father class
def __init__(self, dbsession: Session, user_id: str):
self.user_id = user_id
self.dbsession = dbsession
self.Meta.queryset = self.Meta.queryset.with_session(dbsession)
self.logger = logging.getLogger("REDACTED")
#property
def queryset(self):
return self.Meta.queryset
def get(self, entity_id) -> Base:
entity = self.queryset.filter(self.Meta.model.id == entity_id).first()
if not entity:
raise NoResultsFound(f"Could not find requested ID {entity_id}")
As you can see, the there's already some exception treatment. I really don't see what other exception I could try to catch on AuthService.get
I found the solution to be much simpler than tinkering inside Pyramid or SQLAlchemy.
Debugging my Authentication Policy closely, I found out that my it was keeping a sticky reference for the dbsession. It was stored on the first request ever who used it, and never released.
The first request works as expected, the following one fails: My understanding is that the object is still in memory while the app is running, and after the initial transaction is closed. The second request has a new connection, and a new transaction, but the object in memory still points to the previous one, that when used ultimately causes this.
What I don't understand is why the exception didn't happen sometimes. As I mentioned initially, it was seemingly random.
Another thing that I struggled with was in writing a test case to expose the issue. On my tests, the issue never happens because I have (and I've never seen it done differently) a single connection and a single transaction throughout the entire testing session, as opposed of a new connection/transaction per request, so I have not found no way to actually reproduce.
Please let me know if that makes sense, and if you can shed a light on how to expose the bug on a test case.
I am trying to check if a certain dataset exists in bigquery using the Google Api Client in Python. It always worked untill the last update where I got this strange error I don't know how to fix:
Traceback (most recent call last):
File "/root/miniconda/lib/python2.7/site-packages/dsUtils/bq_utils.py", line 106, in _get
resp = bq_service.datasets().get(projectId=self.project_id, datasetId=self.id).execute(num_retries=2)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/util.py", line 140, in positional_wrapper
return wrapped(*args, **kwargs)
File "/root/miniconda/lib/python2.7/site-packages/googleapiclient/http.py", line 755, in execute
method=str(self.method), body=self.body, headers=self.headers)
File "/root/miniconda/lib/python2.7/site-packages/googleapiclient/http.py", line 93, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 598, in new_request
self._refresh(request_orig)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 864, in _refresh
self._do_refresh_request(http_request)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 891, in _do_refresh_request
body = self._generate_refresh_request_body()
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 1597, in _generate_refresh_req
uest_body
assertion = self._generate_assertion()
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/service_account.py", line 263, in _generate_ass
ertion
key_id=self._private_key_id)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/crypt.py", line 97, in make_signed_jwt
signature = signer.sign(signing_input)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/_pycrypto_crypt.py", line 101, in sign
return PKCS1_v1_5.new(self._key).sign(SHA256.new(message))
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Signature/PKCS1_v1_5.py", line 112, in sign
m = self._key.decrypt(em)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/PublicKey/RSA.py", line 174, in decrypt
return pubkey.pubkey.decrypt(self, ciphertext)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/PublicKey/pubkey.py", line 93, in decrypt
plaintext=self._decrypt(ciphertext)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/PublicKey/RSA.py", line 235, in _decrypt
r = getRandomRange(1, self.key.n-1, randfunc=self._randfunc)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Util/number.py", line 123, in getRandomRange
value = getRandomInteger(bits, randfunc)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Util/number.py", line 104, in getRandomInteger
S = randfunc(N>>3)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 202, in read
return self._singleton.read(bytes)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 178, in read
return _UserFriendlyRNG.read(self, bytes)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 137, in read
self._check_pid()
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 153, in _check_pid
raise AssertionError("PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()")
AssertionError: PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()
Is someone understanding what is hapening?
Note that I also get this error with other bricks like GCStorage.
Note also that I use the following command to load my Google credentials:
from oauth2client.client import GoogleCredentials
def get_credentials(credentials_path): #my json credentials path
logger.info('Getting credentials...')
try:
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = credentials_path
credentials = GoogleCredentials.get_application_default()
return credentials
except Exception as e:
raise e
So if anyone know a better way to load my google credentials using my json service account file, and which would avoid the error, please tell me.
It looks like the error is in the PyCrypto module, which appears to be used under the hood by Google's OAuth2 implementation. If your code is calling os.fork() at some point, you may need to call Crypto.Random.atfork() afterward in both the parent and child process in order to update the module's internal state.
See here for PyCrypto docs; search for "atfork" for more info:
https://github.com/dlitz/pycrypto
This question and answer might also be relevant:
PyCrypto : AssertionError("PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()")
I'm trying to set up a regular expression for grabbing variables from a custom server log. The variables are grabbed by naming each portion of a regular expression. Here's an example:
/^(?<time>[^ ]+) (?<host>[^ ]+) (?<process>[^:]+): (?<message>((?<key>[^ :]+)[ :])? ?((to|from)=<(?<address>[^>]+)>)?.*)$/
I want to do the same for a custom format. The format is as follows:
[11/Jul/2014 19:35:38] ERROR [django.request.tastypie:273] Internal Server Error: /v1/notes/
Traceback (most recent call last):
File "/opt/client-api/venv/local/lib/python2.7/site-packages/tastypie/resources.py", line 195, in wrapper
response = callback(request, *args, **kwargs)
File "/opt/client-api/venv/local/lib/python2.7/site-packages/tastypie/resources.py", line 426, in dispatch_list
return self.dispatch('list', request, **kwargs)
File "/opt/client-api/venv/local/lib/python2.7/site-packages/tastypie/resources.py", line 458, in dispatch
response = method(request, **kwargs)
File "/opt/client-api/venv/local/lib/python2.7/site-packages/tastypie/resources.py", line 1320, in post_list
updated_bundle = self.obj_create(bundle, **self.remove_api_resource_names(kwargs))
File "/opt/client-api/venv/local/lib/python2.7/site-packages/tastypie/resources.py", line 2084, in obj_create
return self.save(bundle)
File "/opt/client-api/venv/local/lib/python2.7/site-packages/tastypie/resources.py", line 2230, in save
bundle.obj.save()
File "./notification/models.py", line 193, in save
handler.handle_notification()
File "./notification/handler.py", line 31, in handle
getattr(self, '_set_{}_payload'.format(preference))()
File "./notification/notification_handler.py", line 89, in _set_payload
raise Exception(error)
All I care about is grabbing the various components on the first line, putting them into variables, and then putting the entire traceback in a variable.
I realize this may be too specific of a question, but I feel a basic run down of the regex that would cover this would be beneficial to many people trying to parse server logs.
You mean something along these lines?
\[(?P<timestamp>.*?)\] (?P<level>\w+) \[(?P<location>.*?)\] (?P<message>.*?)\n(?P<details>.*?)(?=\n\[|$)
See http://regex101.com/r/zS1fN5/1
Works for the given example, but depends on what quirks and exceptions there can be to this log format.