We Keep Coding

What are the things I should avoid in my python Cloud Function to avoid memory leak?

Generic question
My python Cloud Function raises about 0.05 memory error per second - it is invoked about 150 times per second. I get the feeling my function leaves memory residuals behind, which causes its instances to crash once they have dealt with many requests. What are the things you should do or not do so that your function instance doesn't eat "a bit more of its allocated memory" each time it's called ? I've been pointed to the docs to learn that I should delete all temporary files as this is writing in memory but I don't think I've written any.
More context
The code of my function can be summed up as the following.
Global context: Grab a file on Google Cloud Storage containing a list of known User-Agents of bots. Instantiate an Error Reporting client.
If User-Agent identifies a bot, return a 200 code. Else parse the arguments of the request, rename them, format them, timestamp the reception of the request.
Send the resulting message to Pub/Sub in a JSON string.
Return a 200 code
I believe my instances are gradually consuming all memory available because of this graph I've done in Stackdriver:
It is a heat map of the memory usage across my Cloud function's instances, red and yellow indicating that most of my function instances are consuming this range of memory. Because of the cycle that seems to appear, I interpreted it as a gradual fill-up of my instances' memory, until they crash and new instances are spawned. This cycle remains if I raise the memory allocated to the function, it just raises the upper bound of memory usage the cycle follows.
Edit: Code excerpt and more context
The requests contain parameters that help implement tracking on an ecommerce website. Now that I copy it, there might be an anti-pattern where I modify form['products'] while iterating over it, but I don't think it would have anything to do with memory waste ?
from json import dumps
from datetime import datetime
from pytz import timezone
from google.cloud import storage
from google.cloud import pubsub
from google.cloud import error_reporting
from unidecode import unidecode
# this is done in global context because I only want to load the BOTS_LIST at
# cold start
PROJECT_ID = '...'
TOPIC_NAME = '...'
BUCKET_NAME = '...'
BOTS_PATH = '.../bots.txt'
gcs_client = storage.Client()
cf_bucket = gcs_client.bucket(BUCKET_NAME)
bots_blob = cf_bucket.blob(BOTS_PATH)
BOTS_LIST = bots_blob.download_as_string().decode('utf-8').split('\r\n')
del cf_bucket
del gcs_client
del bots_blob
err_client = error_reporting.Client()
def detect_nb_products(parameters):
'''
Detects number of products in the fields of the request.
'''
# ...
def remove_accents(d):
'''
Takes a dictionary and recursively transforms its strings into ASCII
encodable ones
'''
# ...
def safe_float_int(x):
'''
Custom converter to float / int
'''
# ...
def build_hit_id(d):
'''concatenate specific parameters from a dictionary'''
# ...
def cloud_function(request):
"""Actual Cloud Function"""
try:
time_received = datetime.now().timestamp()
# filtering bots
user_agent = request.headers.get('User-Agent')
if all([bot not in user_agent for bot in BOTS_LIST]):
form = request.form.to_dict()
# setting the products field
nb_prods = detect_nb_products(form.keys())
if nb_prods:
form['products'] = [{'product_name': form['product_name%d' % i],
'product_price': form['product_price%d' % i],
'product_id': form['product_id%d' % i],
'product_quantity': form['product_quantity%d' % i]}
for i in range(1, nb_prods + 1)]
useful_fields = [] # list of keys I'll keep from the form
unwanted = set(form.keys()) - set(useful_fields)
for key in unwanted:
del form[key]
# float conversion
if nb_prods:
for prod in form['products']:
prod['product_price'] = safe_float_int(
prod['product_price'])
# adding timestamp/hour/minute, user agent and date to the hit
form['time'] = int(time_received)
form['user_agent'] = user_agent
dt = datetime.fromtimestamp(time_received)
form['date'] = dt.strftime('%Y-%m-%d')
remove_accents(form)
friendly_names = {} # dict to translate the keys I originally
# receive to human friendly ones
new_form = {}
for key in form.keys():
if key in friendly_names.keys():
new_form[friendly_names[key]] = form[key]
else:
new_form[key] = form[key]
form = new_form
del new_form
# logging
print(form)
# setting up Pub/Sub
publisher = pubsub.PublisherClient()
topic_path = publisher.topic_path(PROJECT_ID, TOPIC_NAME)
# sending
hit_id = build_hit_id(form)
message_future = publisher.publish(topic_path,
dumps(form).encode('utf-8'),
time=str(int(time_received * 1000)),
hit_id=hit_id)
print(message_future.result())
return ('OK',
200,
{'Access-Control-Allow-Origin': '*'})
else:
# do nothing for bots
return ('OK',
200,
{'Access-Control-Allow-Origin': '*'})
except KeyError:
err_client.report_exception()
return ('err',
200,
{'Access-Control-Allow-Origin': '*'})

There are a few things you could try (theoretical answer, I didn't play with CFs yet):
explicitly delete the temporary variables that you allocate on the bots processing path, which may be referencing each-other thus preventing the memory garbage collector from freeing them (see https://stackoverflow.com/a/33091796/4495081): nb_prods, unwanted, form, new_form, friendly_names, for example.
if unwanted is always the same make it a global instead.
delete form before re-assigning it to new_form (the old form object remains); also deleting new_form won't actually save much since the object remains referenced by form. I.e. change:
form = new_form
del new_form
into
del form
form = new_form
explicitly invoke the memory garbage collector after you publish your topic and before returning. I'm unsure if that's applicable to CFs or if the invocation is immediately effective or not (for example in GAE it's not, see When will memory get freed after completing the request on App Engine Backend Instances?). This may also be overkill and potentially hurt you CF's performance, see if/how it works for you.
gc.collect()

How to monitor game address values?

After trying to break down code from GitHub and find any youtube videos that talk about this I'm starting to give up, so I'm hoping one of you can please help me. All I want to be able to do is monitor a games memory addresses value. For example, let's say in the game Minecraft the health value and the memory address is:
Address: 001F6498
Value: 20
How do I turn this value into a variable in Python?
Code Thought Process:
import pywin32
pid = 5601
address = 001F6498
ReadProcessMemory(pid, address):
print(Value)
#In this example i would like it to print 20

You need to get a handle to the process first. Here is some code that does so using OpenProcess() FindWindow() and GetWindowThreadProcessId() to get the handle to the process. Also included is a little function to properly read the correct size variable and store it correctly. This method can be used to read pointers, utilizing "i" to denote an integer type.
import win32api
import win32gui
import win32process
from ctypes import *
from pymem import *
PROCESS_ALL_ACCESS = 0x1F0FFF
ReadProcessMemory = windll.kernel32.ReadProcessMemory
def read_memory(procId, address, type):
buffer = (ctypes.c_byte * getlenght(type))()
bytesRead = ctypes.c_ulonglong(0)
readlenght = getlenght(type)
ReadProcessMemory(procId, address, buffer, readlenght, byref(bytesRead))
return struct.unpack(type, buffer)[0]
hWnd = win32gui.FindWindow(0, ("WINDOW NAME HERE"))
pid=win32process.GetWindowThreadProcessId(hWnd)
handle = pymem.Pymem()
handle.open_process_from_id(pid[1])
procBaseAddress = handle.process_base
hProc = windll.kernel32.OpenProcess(PROCESS_ALL_ACCESS, 0, pid[1])
value = ReadProcessMemory(hProc, ADDRESS_OF_VARIABLE_TO_READ, "i")
print(value)
Credits to a friend, puppetmaster, who taught me how to do this

Return a non-repeating, random result from a list inside a function

I am writing a Python program that responds to request and vocalizes a response back to the user. Below is a sample of two functions. How can I do this without using a global variable and still get back a non-repeating, random response?
# stores prior response
website_result = 'first_response.wav'
def launch_website():
# if service is offline return with default msg otherwise launch service
if is_connected() == 'FALSE':
arg = 'this_service_is_offline.wav'
return arg
else:
site = 'http://www.somesite.com'
launch_it(site)
return launch_website_response()
def launch_website_response():
# using the global variable inside function
global website_result
# possible responses
RESPONSES = ['first_response.wav', 'second_response.wav', 'third_response.wav']
# ensures a non-repeating response
tmp = random.choice(RESPONSES)
while website_result == tmp:
tmp = random.choice(RESPONSES)
website_result = tmp
return website_result

Your website_result variable indicates that you to have some sort of persistent state. Maybe you could consider storing it in a text file and access it everytime you need it and change it afterward (this works if you don't have to do too many calls to it, otherwise you will get in I/O limitations).
I don't know about the specifics of your application, but it might happen that you could also make your two functions take website_result as an argument as suggested by #JGut.

couchdb-python change notifications

I'm trying to use couchdb.py to create and update databases. I'd like to implement notification changes, preferably in continuous mode. Running the test code posted below, I don't see how the changes scheme works within python.
class SomeDocument(Document):
#############################################################################
# def __init__ (self):
intField = IntegerField()#for now - this should to be an integer
textField = TextField()
couch = couchdb.Server('http://127.0.0.1:5984')
databasename = 'testnotifications'
if databasename in couch:
print 'Deleting then creating database ' + databasename + ' from server'
del couch[databasename]
db = couch.create(databasename)
else:
print 'Creating database ' + databasename + ' on server'
db = couch.create(databasename)
for iii in range(5):
doc = SomeDocument(intField=iii,textField='somestring'+str(iii))
doc.store(db)
print doc.id + '\t' + doc.rev
something = db.changes(feed='continuous',since=4,heartbeat=1000)
for iii in range(5,10):
doc = SomeDocument(intField=iii,textField='somestring'+str(iii))
doc.store(db)
time.sleep(1)
print something
print db.changes(since=iii-1)
The value
db.changes(since=iii-1)
returns information that is of interest, but in a format from which I haven't worked out how to extract the sequence or revision numbers, or the document information:
{u'last_seq': 6, u'results': [{u'changes': [{u'rev': u'1-9c1e4df5ceacada059512a8180ead70e'}], u'id': u'7d0cb1ccbfd9675b4b6c1076f40049a8', u'seq': 5}, {u'changes': [{u'rev': u'1-bbe2953a5ef9835a0f8d548fa4c33b42'}], u'id': u'7d0cb1ccbfd9675b4b6c1076f400560d', u'seq': 6}]}
Meanwhile, the code I'm really interested in using:
db.changes(feed='continuous',since=4,heartbeat=1000)
Returns a generator object and doesn't appear to provide notifications as they come in, as the CouchDB guide suggests ....
Has anyone used changes in couchdb-python successfully?

I use long polling rather than continous, and that works ok for me. In long polling mode db.changes blocks until at least one change has happened, and then returns all the changes in a generator object.
Here is the code I use to handle changes. settings.db is my CouchDB Database object.
since = 1
while True:
changes = settings.db.changes(since=since)
since = changes["last_seq"]
for changeset in changes["results"]:
try:
doc = settings.db[changeset["id"]]
except couchdb.http.ResourceNotFound:
continue
else:
// process doc
As you can see it's an infinite loop where we call changes on each iteration. The call to changes returns a dictionary with two elements, the sequence number of the most recent update and the objects that were modified. I then loop through each result loading the appropriate object and processing it.
For a continuous feed, instead of the while True: line use for changes in settings.db.changes(feed="continuous", since=since).

I setup a mailspooler using something similar to this. You'll need to also load couchdb.Session() I also use a filter for only receiving unsent emails to the spooler changes feed.
from couchdb import Server
s = Server('http://localhost:5984/')
db = s['testnotifications']
# the since parameter defaults to 'last_seq' when using continuous feed
ch = db.changes(feed='continuous',heartbeat='1000',include_docs=True)
for line in ch:
doc = line['doc']
// process doc here
doc['priority'] = 'high'
doc['recipient'] = 'Joe User'
# doc['state'] + 'sent'
db.save(doc)
This will allow you access your doc directly from the changes feed, manipulate your data as you see fit, and finally update you document. I use a try/except block on the actual 'db.save(doc)' so I can catch when a document has been updated while I was editing and reload the doc before saving.

SUDS - programmatic access to methods and types

I'm investigating SUDS as a SOAP client for python. I want to inspect the methods available from a specified service, and the types required by a specified method.
The aim is to generate a user interface, allowing users to select a method, then fill in values in a dynamically generated form.
I can get some information on a particular method, but am unsure how to parse it:
client = Client(url)
method = client.sd.service.methods['MyMethod']
I am unable to programmaticaly figure out what object type I need to create to be able to call the service
obj = client.factory.create('?')
res = client.service.MyMethod(obj, soapheaders=authen)
Does anyone have some sample code?

Okay, so SUDS does quite a bit of magic.
A suds.client.Client, is built from a WSDL file:
client = suds.client.Client("http://mssoapinterop.org/asmx/simple.asmx?WSDL")
It downloads the WSDL and creates a definition in client.wsdl. When you call a method using SUDS via client.service.<method> it's actually doing a whole lot of recursive resolve magic behind the scenes against that interpreted WSDL. To discover the parameters and types for methods you'll need to introspect this object.
For example:
for method in client.wsdl.services[0].ports[0].methods.values():
print '%s(%s)' % (method.name, ', '.join('%s: %s' % (part.type, part.name) for part in method.soap.input.body.parts))
This should print something like:
echoInteger((u'int', http://www.w3.org/2001/XMLSchema): inputInteger)
echoFloatArray((u'ArrayOfFloat', http://soapinterop.org/): inputFloatArray)
echoVoid()
echoDecimal((u'decimal', http://www.w3.org/2001/XMLSchema): inputDecimal)
echoStructArray((u'ArrayOfSOAPStruct', http://soapinterop.org/xsd): inputStructArray)
echoIntegerArray((u'ArrayOfInt', http://soapinterop.org/): inputIntegerArray)
echoBase64((u'base64Binary', http://www.w3.org/2001/XMLSchema): inputBase64)
echoHexBinary((u'hexBinary', http://www.w3.org/2001/XMLSchema): inputHexBinary)
echoBoolean((u'boolean', http://www.w3.org/2001/XMLSchema): inputBoolean)
echoStringArray((u'ArrayOfString', http://soapinterop.org/): inputStringArray)
echoStruct((u'SOAPStruct', http://soapinterop.org/xsd): inputStruct)
echoDate((u'dateTime', http://www.w3.org/2001/XMLSchema): inputDate)
echoFloat((u'float', http://www.w3.org/2001/XMLSchema): inputFloat)
echoString((u'string', http://www.w3.org/2001/XMLSchema): inputString)
So the first element of the part's type tuple is probably what you're after:
>>> client.factory.create(u'ArrayOfInt')
(ArrayOfInt){
_arrayType = ""
_offset = ""
_id = ""
_href = ""
_arrayType = ""
}
Update:
For the Weather service it appears that the "parameters" are a part with an element not a type:
>>> client = suds.client.Client('http://www.webservicex.net/WeatherForecast.asmx?WSDL')
>>> client.wsdl.services[0].ports[0].methods.values()[0].soap.input.body.parts[0].element
(u'GetWeatherByZipCode', http://www.webservicex.net)
>>> client.factory.create(u'GetWeatherByZipCode')
(GetWeatherByZipCode){
ZipCode = None
}
But this is magic'd into the parameters of the method call (a la client.service.GetWeatherByZipCode("12345"). IIRC this is SOAP RPC binding style? I think there's enough information here to get you started. Hint: the Python command line interface is your friend!

According to suds documentation, you can inspect service object with __str()__. So the following gets a list of methods and complex types:
from suds.client import Client;
url = 'http://www.webservicex.net/WeatherForecast.asmx?WSDL'
client = Client(url)
temp = str(client);
The code above produces following result (contents of temp):
Suds ( https://fedorahosted.org/suds/ ) version: 0.3.4 (beta) build: R418-20081208
Service ( WeatherForecast ) tns="http://www.webservicex.net"
Prefixes (1)
ns0 = "http://www.webservicex.net"
Ports (2):
(WeatherForecastSoap)
Methods (2):
GetWeatherByPlaceName(xs:string PlaceName, )
GetWeatherByZipCode(xs:string ZipCode, )
Types (3):
ArrayOfWeatherData
WeatherData
WeatherForecasts
(WeatherForecastSoap12)
Methods (2):
GetWeatherByPlaceName(xs:string PlaceName, )
GetWeatherByZipCode(xs:string ZipCode, )
Types (3):
ArrayOfWeatherData
WeatherData
WeatherForecasts
This would be much easier to parse. Also every method is listed with their parameters along with their types. You could, probably, even use just regular expression to extract information you need.

Here's a quick script I wrote based on the above information to list the input methods suds reports as available on a WSDL. Pass in the WSDL URL. Works for the project I'm currently on, I can't guarantee it for yours.
import suds
def list_all(url):
client = suds.client.Client(url)
for service in client.wsdl.services:
for port in service.ports:
methods = port.methods.values()
for method in methods:
print(method.name)
for part in method.soap.input.body.parts:
part_type = part.type
if(not part_type):
part_type = part.element[0]
print(' ' + str(part.name) + ': ' + str(part_type))
o = client.factory.create(part_type)
print(' ' + str(o))

You can access suds's ServiceDefinition object. Here's a quick sample:
from suds.client import Client
c = Client('http://some/wsdl/link')
types = c.sd[0].types
Now, if you want to know the prefixed name of a type, it's also quite easy:
c.sd[0].xlate(c.sd[0].types[0][0])
This double bracket notation is because the types are a list (hence a first [0]) and then in each item on this list there may be two items. However, suds's internal implementation of __unicode__ does exactly that (i.e. takes only the first item on the list):
s.append('Types (%d):' % len(self.types))
for t in self.types:
s.append(indent(4))
s.append(self.xlate(t[0]))
Happy coding ;)

Once you created WSDL method object you can get information about it from it's __metadata__, including list of it's arguments' names.
Given the argument's name, you can access it's actual instance in the method created. That instance also contains it's information in __metadata__, there you can get it's type name
# creating method object
method = client.factory.create('YourMethod')
# getting list of arguments' names
arg_names = method.__metadata__.ordering
# getting types of those arguments
types = [method.__getitem__(arg).__metadata__.sxtype.name for arg in arg_names]
Disclaimer: this only works with complex WSDL types. Simple types, like strings and numbers, are defaulted to None

from suds.client import Client
url = 'http://localhost:1234/sami/2009/08/reporting?wsdl'
client = Client(url)
functions = [m for m in client.wsdl.services[0].ports[0].methods]
count = 0
for function_name in functions:
print (function_name)
count+=1
print ("\nNumber of services exposed : " ,count)

i needed an example of using suds with objects.
beside the answers found here, i found a very good article
that answered my question even further.
here is a short summary:
first, print the client to see an overview of it's content.
from suds.client import Client client =
Client("https://wsvc.cdiscount.com/MarketplaceAPIService.svc?wsdl")
print client
second, create an instance of a type (using it's name including it's prefix ns*.), and print it, to see it's member data.
HeaderMessage = client.factory.create('ns0:HeaderMessage')
print HeaderMessage
to fill your object's data members, either assign them a scalar value for scalar members, or a dict, to object members.
HeaderMessage.Context = {
"CatalogID": "XXXXX"
"CustomerID": 'XXXXX'
"SiteID": 123
}
members whose type name start with ArrayOf expect a list of objects of the type mentioned in the rest of the type name.
ArrayOfDomainRights = client.factory.create('ns0:ArrayOfDomainRights')
ArrayOfDomainRights.DomainRights = [XXXXXXXXXXXXX, XXXXXXXXXXXX]

i needed an example of using suds with objects.
beside the answers found here, i found a very good article
that answered my question even further.
here is a short summary:
first, print the client to see an overview of it's content.
from suds.client import Client client =
Client("https://wsvc.cdiscount.com/MarketplaceAPIService.svc?wsdl")
print client
second, create an instance of a type (using it's name including it's prefix ns*.), and print it, to see it's member data.
HeaderMessage = client.factory.create('ns0:HeaderMessage')
print HeaderMessage
to fill your object's data members, either assign them a scalar value for scalar members, or a dict, to object members.
HeaderMessage.Context = {
"CatalogID": "XXXXX"
"CustomerID": 'XXXXX'
"SiteID": 123
}
members whose type name start with ArrayOf expect a list of objects of the type mentioned in the rest of the type name.
ArrayOfDomainRights = client.factory.create('ns0:ArrayOfDomainRights')
ArrayOfDomainRights.DomainRights = [XXXXXXXXXXXXX, XXXXXXXXXXXX]