A simple filesystem cache for python.
The phyles home page is at http://pyfscache.bravais.net and the source code is maintained at github: https://github.com/jcstroud/pyfscache/.
Pyfscache (python filesystem cache) is a filesystem cache that is easy to use. The principal class is pyfscache.FSCache, instances of which may be used as decorators to create cached functions with very little coding overhead:
import pyfscache
cache_it = pyfscache.FSCache('some/cache/directory',
days=13, hours=4, minutes=2.5)
@cache_it
def cached_doit(a, b, c):
return [a, b, c]
And that’s it!
Now, every time the function cached_doit() is called with a particular set of arguments, the cache cache_it is inspected to see if an identical call has been made before. If it has, then the return value is retrieved from the cache_it cache. If not, the return value is calculated with cached_doit(), stored in the cache, and then returned.
In the code above, the expiration for the cache is set to 1137750 seconds (13 days, 4 hours, and 2.5 minutes). Values may be provided for years, months, weeks, days, hours, minutes, and seconds. The time is the total for all keywords.
If these optional keyword arguments are not included, then items added by the pyfscache.FSCache object never expire:
no_expiry_cache = pyfscache.FSCache('some/cache/directory')
Note
Several instances of pyfscache.FSCache objects can use the same cache directory. Each will honor the expirations of the items therein. Thus, it is possible to have a cache mixed with objects of many differening lifetimes, made by many instances of pyfscache.FSCache.
It is not necessary to use decorators, although their convenience is manifest in the example above:
import pyfscache
cache = pyfscache.FSCache('some/cache/directory',
days=13, hours=4, minutes=2.5)
def uncached_doit(a, b, c):
return [a, b, c]
cached_doit = cache(uncached_doit)
pyfscache: A file system cache for python. Copyright (c) 2013, James C. Stroud; All rights reserved.
A class that manages a filesystem cache. Works like a dictionary and can decorate functions to make them cached:
>>> import os
>>> import shutil
>>> from pyfscache import *
>>> if os.path.exists('erase/me'):
... shutil.rmtree('erase/me')
...
>>> c = FSCache('erase/me', days=7)
>>> c['some_key'] = "some_value"
>>> c['some_key']
'some_value'
>>> os.listdir('erase/me')
['PXBZzwEy3XnbOweuMtoPj9j=PwkfAsTXexmW2v05JD']
>>> c.expire('some_key')
>>> os.listdir('erase/me')
[]
>>> c['some_key'] = "some_value"
>>> @c
... def doit(avalue):
... print "had to call me!"
... return "some other value"
...
>>> doit('some input')
had to call me!
'some other value'
>>> doit('some input')
'some other value'
>>> shutil.rmtree('erase/me')
Unloads all loaded cache items from memory. All cache items remain on the disk, however.
Use with care. This permanently removes the object keyed by k from the cache, both in the memory and in the filesystem.
Returns an expiry for the cache in seconds as if the start of the expiration period were the moment at which this the expiry() function is called.
Returns the lifetime, in seconds, of new items in the cache. If new items do not expire, then None is returned.
Returns a list of keys for all objects that are loaded.
Returns the names of the files in the cache on the filesystem. These are not keys but one-way hashes (or “digests”) of the keys (created by make_digest()).
Returns the absolute path to the file system cache represented by the instance.
Returns True if the item keyed by k has been loaded, False if not.
Returns the lifetime, in seconds, of new items in the cache. If new items do not expire, then None is returned.
Causes the object keyed by k to be loaded from the file system and returned. It therefore causes this object to reside in memory.
Returns the absolute path to the file system cache represented by the instance.
Be careful, this empties the cache from both the filesystem and memory!
Removes the object keyed by k from memory but not from the filesystem. To remove it from both memory and permanently from the filesystem, use expire.
Use with care. Updates, both in memory and on the filesystem, the object for key k with the object v. If the key k already exists with a stored object, it will be replaced.
Creates a digest suitable for use within an FSCache object from the key object k.
>>> adict = {'a' : {'b':1}, 'f': []}
>>> make_digest(adict)
'a2VKynHgDrUIm17r6BQ5QcA5XVmqpNBmiKbZ9kTu0A'
Creates a cached function from function f. The cache can be any mapping object, such as FSCache objects.
The function arguments are expected to be well-behaved for python’s cPickle. Or, in other words, the expected values for the parameters (the arguments) should be instances new-style classes (i.e. inheriting from object) or implement __getstate__() with well-behaved results.
If the arguments to f are not expected to be well-behaved, it is best to use cache_function instead and create a custom keyer.
Takes any function f and a function that creates a key, keyer and caches the result in cache.
The keys created by keyer should be well behaved for python’s cPickle. See the documentation for auto_cache_funtion() for details.
It is best to have a unique keyer for every function.
Converts keyword arguments to seconds.
The the keyword arguments can have the following keys:
- years (31,556,900 seconds per year)
- months (2,629,740 seconds per month)
- weeks (604,800 seconds per week)
- days (86,400 seconds per day)
- hours (3600 seconds per hour)
- minutes (60 seconds per minute)
- seconds
>>> to_seconds(seconds=15, minutes=20)
1215.0
>>> to_seconds(seconds=15.42, hours=10, minutes=18, years=2)
63150895.42