I'm trying to understand how Django is setting keys for my views. I'm wondering if there's a way to just get all the saved keys from Memcached. something like a cache.all() or something. I've been trying to find the key with cache.has_key('test') but still cant figure out how the view keys are being named.
UPDATE: The reason I need this is because I need to manually delete parts of the cache but dont know the key values Django is setting for my cache_view key
For RedisCache you can get all available keys with.
from django.core.cache import cache
cache.keys('*')
As mentioned there is no way to get a list of all cache keys within django. If you're using an external cache (e.g. memcached, or database caching) you can inspect the external cache directly.
But if you want to know how to convert a django key to the one used in the backend system, django's make_key() function will do this.
https://docs.djangoproject.com/en/1.8/topics/cache/#cache-key-transformation
>>> from django.core.cache import caches
>>> caches['default'].make_key('test-key')
u':1:test-key'
For debugging, you can temporarily switch to LocMemCache instead of PyMemcacheCache:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'unique-snowflake',
}
}
then see this question:
from django.core.cache.backends import locmem
print(locmem._caches)
In my setup with Django 3.2 there is a method to get a "raw" client for Redis which you can get the keys from.
from django.core.cache import cache
cache.get_client(1).keys()
You can use http://www.darkcoding.net/software/memcached-list-all-keys/ as explained in How do I check the content of a Django cache with Python memcached?
The Memcached documentation recommends that instead of listing all the cache keys, you run memcached in verbose mode and see everything that gets changed. You should start memcached like this
memcached -vv
and then it will print the keys as they get created/updated/deleted.
For Redis Backend
I'm going to add this answer because I landed on this SO question searching for exactly the same question but using a different cache backend. Also with REDIS in particular if you are using the same REDIS server for multiple applications you will want to scope your cache keys with the KEY_PREFIX option otherwise you could end up with cache keys from another application.
My answer is for if you have setup KEY_PREFIX in your settings.py and if you are using either redis_cache.RedisCache or django.core.cache.backends.redis.RedisCache
e.g.
CACHES = {
"default": {
"BACKEND": "redis_cache.RedisCache",
"LOCATION": f"redis://localhost:6379",
"KEY_PREFIX": "my_prefix",
},
}
or
CACHES = {
"default": {
"BACKEND": "django.core.cache.backends.redis.RedisCache",
"LOCATION": f"redis://localhost:6379",
"KEY_PREFIX": "my_prefix",
},
}
redis_cache.RedisCache
from django.conf import settings
from django.core.cache import cache
cache_keys = cache.get_client(1).keys(
f"*{settings.CACHES['default']['KEY_PREFIX']}*"
)
django.core.cache.backends.redis.RedisCache
Doing some tests shows that using Django's built in RedisCache may already be scoped but in my case I'm doing it to be explicit. calling .keys("*") will also return keys that belong to celery tasks
from django.conf import settings
from django.core.cache import cache
cache_keys = cache._cache.get_client().keys(
f"*{settings.CACHES['default']['KEY_PREFIX']}*"
)
Bonus: Deleting all app keys
If you want to clear the cache for your specific app instead of ALL the keys in REDIS you'll want to using the prior technique and then call cache.delete_many(cache_keys) instead of cache.clear() as the Django Docs warns that using cache.clear() will remove ALL keys in your cache, not just the ones that are created by your app.
You can use memcached_stats from: https://github.com/dlrust/python-memcached-stats. This package makes it possible to view the memcached keys from within the python environment.
If this is not too out of date, I have had similar issue, due I have had to iterate over whole cache. I managed it, when I add something to my cache like in following pseudocode:
#create caches key list if not exists
if not my_cache.get("keys"):
my_cache.set("keys", [])
#add to my cache
my_cache.set(key, value)
#add key to keys
if key not in my_cache.get("keys"):
keys_list = my_cache.get("keys")
keys_list.append(key)
my_cache.set("keys", keys_list)
this helps.
Ref:
https://lzone.de/blog/How-to%20Dump%20Keys%20from%20Memcache
https://github.com/dlrust/python-memcached-stats
import re, telnetlib, sys
key_regex = re.compile(r"ITEM (.*) \[(.*); (.*)\]")
slab_regex = re.compile(r'STAT items:(.*):number')
class MemcachedStats:
def __init__(self, host='localhost', port='11211'):
self._host = host
self._port = port
self._client = None
#property
def client(self):
if self._client is None:
self._client = telnetlib.Telnet(self._host, self._port)
return self._client
def command(self, cmd):
' Write a command to telnet and return the response '
self.client.write("{}\n".format(cmd).encode())
res = self.client.read_until('END'.encode()).decode()
return res
def slab_ids(self):
' Return a list of slab ids in use '
slab_ids = slab_regex.findall(self.command('stats items'))
slab_ids = list(set(slab_ids))
return slab_ids
def get_keys_on_slab(self, slab_id, limit=1000000):
cmd = "stats cachedump {} {}".format(slab_id, limit)
cmd_output = self.command(cmd)
matches = key_regex.findall(cmd_output)
keys = set()
for match_line in matches:
keys.add(match_line[0])
return keys
def get_all_keys(self):
slab_ids = self.slab_ids()
all_keys = set()
for slab_id in slab_ids:
all_keys.update(self.get_keys_on_slab(slab_id))
return list(all_keys)
def main():
m = MemcachedStats()
print(m.get_all_keys())
if __name__ == '__main__':
main()
There are some weird workarounds you can do to get all keys from the command line, but there is no way to do this with memcached inside of Django. See this thread.
Related
I need to access defines and variables from the Kamailio configuration file in my Python script. So far, I am able to access the global variables only through self.my_var = int(KSR.pv.get("$sel(cfg_get.my_group.my_var)")) where this variable is defined in the configuration file as my_group.my_var = 664. How can I access the definitions (to know that #!ifdef MY_DEFINE succeeds or not)? Or at least the configuration file to read them myself ?
I found nothing about on the official documentation and even $sel(cfg_get.my_group.my_var) I found it elsewhere.
UPDATE These values are not normally available at runtime (preprocessing), so the native code can use them like this:
#!define WITH_SIPTRACE
#!substdef "!SERVER_ID!654!g"
...
request_route {
#!ifdef WITH_SIPTRACE
xlog("L_NOTICE", "$C(yx)========= server_id=SERVER_ID NEW $rm $pr:$si:$sp$C(xx)\n");
#!endif
...
Can this same behavior be achieved in Python ?
UPDATE 2 Found partial answer (below) in KSR.kx.get_def() and KSR.kx.get_defn().
You should not change defines. There is no mechanism for that.
To make configurable parameters, use database or curl module + in-memory cache via htable module.
Here is htable optimization for auth, for example
AUTH WITH CACHING
# authentication with password caching using htable
modparam("htable", "htable", "auth=>size=10;autoexpire=300;")
modparam("auth_db", "load_credentials", "$avp(password)=password")
route[AUTHCACHE] {
if($sht(auth=>$au::passwd)!=$null) {
if (!pv_auth_check("$fd", "$sht(auth=>$au::passwd)", "0", "1")) {
auth_challenge("$fd", “1”);
exit;
}
} else {
# authenticate requests
if (!auth_check("$fd", "subscriber", "1")) {
auth_challenge("$fd", "0");
exit;
}
$sht(auth=>$au::passwd) = $avp(password);
}
# user authenticated - remove auth header
if(!is_method("REGISTER|PUBLISH"))
consume_credentials();
}
Here is how to get info from db using sql.
https://kamailio.org/docs/modules/5.0.x/modules/avpops.html#avpops.f.avp_db_query
avp_db_query("select password, ha1 from subscriber where username='$tu'",
"$avp(i:678);$avp(i:679)");
As opensource sample of such optimization you can see code of kazoo project.
A solution I dislike because it adds extra variables for each interesting definition.
On the configuration side:
####### Custom Parameters #########
/*
Is there a better method to access these flags (!define)?
For the constants (!substdef) there are KSR.kx.get_def(), KSR.kx.get_defn().
*/
#!ifdef WITH_OPTIONA
my_group.option_a = yes
#!else
my_group.option_a = no
#!endif
#!ifdef WITH_OPTIONB
my_group.option_b = yes
#!else
my_group.option_b = no
#!endif
On the Python side:
def __init__(self):
self.my_domain = KSR.kx.get_def("MY_DOMAIN")
self.server_id = KSR.kx.get_defn("SERVER_ID")
assert self.my_domain and self.server_id
self.flags = None
self.initialized = False
def real_init(self):
'''
Object data is initialized here, dynamically,
because__init__ and child_init are called too early.
'''
def read_flags():
flags = {}
for i in ['option_a', 'option_b']:
value = bool(int(KSR.pv.get("$sel(cfg_get.my_group.{})".format(i))))
if value:
flags[i[len('option_'):]] = True
return flags
self.flags = read_flags()
self.initialized = True
Update Partial solution found: !substdef constants, but not !define ones.
Update 5.5.2 KSR.kx.ifdef() and KSR.kx.ifndef() added on the master branch, but not yet exported in a official release (the latest being 5.5.2)
See kemix: added KSR.kx.ifdef() and KSR.kx.ifndef().
I want to move from Ansible to Nornir. In Ansbile I use dynamic inventory, where I use this python script to reference the host_var folder:
import json
import yaml
import glob
groups = {}
for hostfilename in glob.glob('./host_vars/*.yml'):
with open(hostfilename, 'r') as hostfile:
host = yaml.load(hostfile, Loader=yaml.FullLoader)
for hostgroup in host['host_groups']:
if hostgroup not in groups.keys():
groups[ hostgroup ] = { 'hosts': [] }
groups[ hostgroup ]['hosts'].append( host['hostname'] )
print(json.dumps(groups))
Question:
How can I use my existing Ansible Inventory in Nornir.
nornir.plugins.inventory.ansible.AnsibleInventory can only be used with 1x host.yaml file not with many, at least this is my understanding
Edit: Goal is to create always new Inventory files on every run. The workflow would be to generate the inventory yaml files in host_vars and then use it during the play.
Can somebody please help me?
Thanks
F.
If I understood you correctly, you want each yaml file in the host_vars folder to be interpreted as one host and its data. This feature is not part of base Nornir, but can be implemented via a custom inventory plugin.
The custom inventory plugin should implement a load() method that returns an Inventory-type object that Nornir can then use normally (see here for an example of the SimpleInventory implementation). I came up with this snippet adapted from the code that was given:
import os
import yaml
import glob
import pathlib
from nornir.core.inventory import (
Inventory,
Hosts,
Host,
Groups,
Group)
def map_host_data(host_dict):
return({
'hostname' : host_dict['hostname'],
'port': host_dict.get('port',22),
'username' : host_dict['username'],
'password' : host_dict['password'],
'platform' : host_dict['platform'],
'data' : host_dict.get('data', None)
})
class DynamicInventory:
def __init__(self, inventory_dir: str = "host_vars/") -> None:
self.inventory_dir = pathlib.Path(inventory_dir).expanduser()
def load(self):
hosts = Hosts()
groups = Groups()
for hostfilename in glob.glob(f"{self.inventory_dir}/*.yaml"):
with open(hostfilename,'r') as hostfile:
host_name = os.path.basename(hostfilename).replace('.yaml','')
host = yaml.load(hostfile, Loader=yaml.FullLoader)
for hostgroup in host['host_groups']:
if hostgroup not in groups.keys():
group = Group(name=hostgroup)
groups[hostgroup] = group
hosts[host_name] = Host(name=host_name, **map_host_data(host))
return Inventory(hosts=hosts,groups=groups,defaults={})
I'm assuming you're using Nornir >= 3 (which you really should), so don't forget to register your plugin if using it on your configuration. Assuming you put the above code under plugins/inventory.py:
from nornir import InitNornir
from plugins.inventory import DynamicInventory
from nornir.core.plugins.inventory import InventoryPluginRegister
InventoryPluginRegister.register("DynamicInventoryPlugin",DynamicInventory)
nr = InitNornir(inventory={'plugin': 'DynamicInventoryPlugin'},
runner={'plugin': 'threaded','options': {'num_workers': 20}})
This of course ignores some features (such as setting defaults), but can be modified to add more features that better match your current setup.
Using python pyramid and ElastiSearch. I looked at pythonelasticsearch-dsl which offers a nice ORM but I'm not sure how to integrate it with pyramid.
So far I made a "global connection" as per pythonelasticsearch-dsl and expose the connection via an attribute into pyramid's request.
Do you see anything wrong with this code ?!
from elasticsearch_dsl import connections
def _create_es_connection(config):
registry = config.registry
settings = registry.settings
es_servers = settings.get('elasticsearch.' + 'servers', ['localhost:9200'])
es_timeout = settings.get('elasticsearch.' + 'timeout', 20)
registry.es_connection = connections.create_connection(
hosts=es_servers,
timeout=es_timeout)
def get_es_connection(request):
return getattr(request.registry, 'es_connection',
connections.get_connection())
# main
def main(global_config, **settings):
...
config = Configurator(settings=settings)
config.add_request_method(
get_es_connection,
'es',
reify=True)
I use the connection as
#view
request.es ...
If there are any other ways I would appreciate any pointers - thank you.
A few things look weird, but I guess it comes from the copy/paste from your project (missing type cast in settings, connections undefined, etc.)
What you are trying to do is very similar to what you'd do with SQLAlchemy:
https://docs.pylonsproject.org/projects/pyramid_cookbook/en/latest/database/sqlalchemy.html
But according the docs of pythonelasticsearch-dsl you don't even have to bother with all that, since the lib allows you define a global default connection:
https://elasticsearch-dsl.readthedocs.io/en/latest/configuration.html#default-connection
In my settings.py , I have specified my cache as :
CACHES = {
'default': {
......
}
}
In my views.py, I have
import requests
from django.core.cache import cache, get_cache
def aview():
#check cache
if not get_cache('default').get('key'):
#make request and save in cache
result = request.get('some_url')
get_cache('default').set('key', result)
return result
else:
return get_cache('default').get('key')
Now in my tests.py, I have been able to mock requests.get('aurl'), so that makes sure that no external requests are made.
But the test code still hits the cache and gets/sets from it. So if my prod has already set the cache, then test is failing because it gets the data from same cache. Or if I run my tests first, then the test case is setting the cache with test data and I see that same reflected when I run prod website.
How can I mock the calls to
get_cache('default').set('key', result)
and
get_cache('default').get('key')
so that the set call does not sets the real cache ( return None?) and get does not return anything in actual cache.
Please provide me with code sample to how to get this done.
Here is how I have mocked my requests.get
def test_get_aview(self):
with mock.patch('requests.get') as mymock:
mymock.side_effect = (lambda url: MOCKED_DATA[url])
What code can I put after this to make it work? I tried something like
class MockCacheValue(mock.MagicMock):
def get(self, key):
print 'here'
return None
def set(self, key, value):
print 'here 2'
pass
def test_get_aview(self):
with mock.patch('requests.get') as mymock:
mymock.side_effect = (lambda url: MOCKED_DATA[url])
mock.patch('django.core.cache.get_cache', new=MockCacheValue)
but it does not work and putting a print statement inside get/set above does not print anything giving me an idea that its not mocked properly
I think you should use dummy cache while running tests by:
overriding settings in test cases, see docs
checking what cache backend to use while testing right in settings.py:
CACHES = ...
if 'test' in sys.argv:
CACHES['default'] = {'BACKEND': 'django.core.cache.backends.dummy.DummyCache',}
having a separate settings.py for testing
mocking, see good article on how to do it
Hope that helps.
I want to generate for my users temporary download link.
Is that ok if i use django to generate link using url patterns?
Could it be correct way to do that. Because can happen that I don't understand some processes how it works. And it will overflow my memory or something else. Some kind of example or tools will be appreciated. Some nginx, apache modules probably?
So, what i wanna to achieve is to make url pattern which depend on user and time. Decript it end return in view a file.
A simple scheme might be to use a hash digest of username and timestamp:
from datetime import datetime
from hashlib import sha1
user = 'bob'
time = datetime.now().isoformat()
plain = user + '\0' + time
token = sha1(plain)
print token.hexdigest()
"1e2c5078bd0de12a79d1a49255a9bff9737aa4a4"
Next you store that token in a memcache with an expiration time. This way any of your webservers can reach it and the token will auto-expire. Finally add a Django url handler for '^download/.+' where the controller just looks up that token in the memcache to determine if the token is valid. You can even store the filename to be downloaded as the token's value in memcache.
Yes it would be ok to allow django to generate the urls. This being exclusive from handling the urls, with urls.py. Typically you don't want django to handle the serving of files see the static file docs[1] about this, so get the notion of using url patterns out of your head.
What you might want to do is generate a random key using a hash, like md5/sha1. Store the file and the key, datetime it's added in the database, create the download directory in a root directory that's available from your webserver like apache or nginx... suggest nginx), Since it's temporary, you'll want to add a cron job that checks if the time since the url was generated has expired, cleans up the file and removes the db entry. This should be a django command for manage.py
Please note this is example code written just for this and not tested! It may not work the way you were planning on achieving this goal, but it works. If you want the dl to be pw protected also, then look into httpbasic auth. you can generate and remove entries on the fly in a httpd.auth file using htpasswd and the subprocess module when you create the link or at registration time.
import hashlib, random, datetime, os, shutil
# model to hold link info. has these fields: key (charfield), filepath (filepathfield)
# datetime (datetimefield), url (charfield), orgpath (filepathfield of the orignal path
# or a foreignkey to the files model.
from models import MyDlLink
# settings.py for the app
from myapp import settings as myapp_settings
# full path and name of file to dl.
def genUrl(filepath):
# create a onetime salt for randomness
salt = ''.join(['{0}'.format(random.randrange(10) for i in range(10)])
key = hashlib('{0}{1}'.format(salt, filepath).hexdigest()
newpath = os.path.join(myapp_settings.DL_ROOT, key)
shutil.copy2(fname, newpath)
newlink = MyDlink()
newlink.key = key
newlink.date = datetime.datetime.now()
newlink.orgpath = filepath
newlink.newpath = newpath
newlink.url = "{0}/{1}/{2}".format(myapp_settings.DL_URL, key, os.path.basename(fname))
newlink.save()
return newlink
# in commands
def check_url_expired():
maxage = datetime.timedelta(days=7)
now = datetime.datetime.now()
for link in MyDlink.objects.all():
if(now - link.date) > maxage:
os.path.remove(link.newpath)
link.delete()
[1] http://docs.djangoproject.com/en/1.2/howto/static-files/
It sounds like you are suggesting using some kind of dynamic url conf.
Why not forget your concerns by simplifying and setting up a single url that captures a large encoded string that depends on user/time?
(r'^download/(?P<encrypted_id>(.*)/$', 'download_file'), # use your own regexp
def download_file(request, encrypted_id):
decrypted = decrypt(encrypted_id)
_file = get_file(decrypted)
return _file
A lot of sites just use a get param too.
www.example.com/download_file/?09248903483o8a908423028a0df8032
If you are concerned about performance, look at the answers in this post: Having Django serve downloadable files
Where the use of the apache x-sendfile module is highlighted.
Another alternative is to simply redirect to the static file served by whatever means from django.