Coding N Stuff

Luckily, I’ve been working with python in my last project. I say luckily because python is a language I happen to love.

In this project I had some repetitive code that I hate and goes against many programming philosophies. The problem was with some functions and methods with a couple of arguments that had computed defaults. In my particular case the defaults where current timestamps or value that needed to be retrieved by the database. So I’ve managed to find a great solution for that, using the power of introspection and a feature available since 2.4, decorators.

If you don’t know what decorators are, here’s a quick brief. Decorators are mainly syntactic sugar, that allows us to specify that a given function or method needs to be transformed by other function after it’s creation. There was a lot of discussion around the syntax to use in this new feature, see this PEP, but I think the right decision was taken, since it makes the code more clear.

So the idea behind my solution is that for each function that takes a computed default, we can use a callable as the default, and then the decorated version of the function checks first if that argument was passed, otherwise it computes it using the callable default. Here’s my first draft:

import inspect
 
def dynargs(fn):
    def decorated(*args, **kwargs):
        (argnames, varargs, keywords, defaults) = inspect.getargspec(fn)
        strip = max(len(args), len(argnames) - len(defaults))
 
        for name, default in zip(argnames[strip:], defaults):
            if callable(default) and not name in kwargs:
                kwargs[name] = default()
 
        return fn(*args, **kwargs)
 
    return decorated

The inspect.getargspec function retrieves all there is to know about the function arguments.  Next, we check how many arguments we want to strip out of the list, either the total non named args, because the argument we want to parsed may have been passed as a positional argument, or all the arguments with default values.

The for loop then iterates on a generated list of tuples with name / default pairs (almost like a dictionary), and if the default is callable and it was not passed as a named argument, we add it to the dictionary of named arguments. Here’s a small test:

import datetime
 
@dynargs
def test_func(d=datetime.datetime.now):
    print(d)
 
if __name__ == "__main__":
    import time
    test_func()
    time.sleep(2)
    test_func()
 
#output
#2010-01-14 11:48:43.982000
#2010-01-14 11:48:45.982000

As expected, each call to the test function prints the time at the function call. But as any hack we must be concerned with performance. So I ran a small timeit test:

import datetime
 
@dynargs
def test_func(d=datetime.datetime.now):
    pass
 
def test_func1(d=None):
    if d is None:
        d = datetime.datetime.now()
 
if __name__ == "__main__":
    import timeit
    t = timeit.Timer("test_func()", "from dynargs import test_func")
    print("test_func: %f seconds" % (t.timeit(100000)))
 
    t = timeit.Timer("test_func1()", "from dynargs import test_func1")
    print("test_func1: %f seconds" % (t.timeit(100000)))
 
#output
#test_func: 2.273624 seconds
#test_func1: 0.347172 seconds

As you can see, this first draft can be a real performance killer if you happen to call that particular function a good amount of times through the life cycle of the script. So I decided to try a more elegant solution, which is to create an à la carte decorated function:

import inspect
 
def dynargs(fn):
    (argnames, varargs, keywords, defaults) = inspect.getargspec(fn)
 
    callables = [
        (name, index + 1, default) for ((index, name), default)
        in zip(enumerate(argnames[-len(defaults):]), defaults)
        if callable(default)
    ]
 
    def decorated(*args, **kwargs):
        for name, default in [(name, default) for (name, index, default)
                                in callables if index > len(args) and not name in kwargs]:
            kwargs[name] = default()
 
        return fn(*args, **kwargs)
 
    return decorated

Looks weird because it as some complicated list comprehensions here (I love python list comprehensions), but is not hard to understand. Callables becomes a list of tuples in the form of containing each the name, index position on the argument list and obviously, the callable default. Then the decorated function adds all non-passed arguments in the callables list if their indexes are higher than the number of element on the passed positional arguments list and are not passed as named arguments.  Now all the introspection is made at compile time and not at run time. A new timeit test shows the following results:

#output
#test_func: 0.588709 seconds
#test_func1: 0.312928 seconds

That’s way better, but this is a very generic decorator, and can get even better if you adapt it to your particular situation, for example, you may not need the index of each callable default if you know in advance that you are going to use only named arguments when calling functions decorated by dynargs, thus you can skip some computations.

Now let’s play with this a bit. We know that python decorators can be a call to a function that returns another decorator. Let’s imagine the following situation, we have a couple of functions that have arguments that if not present should be pulled out of a database for example. We can create a decorator for this specific case:

import inspect
 
def get_that_setting(name):
    return "setting is %s" % name
 
def db_setting(argname, setting):
    def decorator(fn):
        index = [index for (index, name) in enumerate(inspect.getargspec(fn)[0]) if name == argname][0]
 
        def decorated(*args, **kwargs):
            if index > len(args) - 1 and not argname in kwargs:
                kwargs[argname] = get_that_setting(setting)
 
            return fn(*args, **kwargs)
 
        return decorated
 
    return decorator
 
@db_setting("some_setting", "name_ind_db")
def test_func(some_setting):
    print(some_setting)
 
if __name__ == "__main__":
    test_func()
    test_func("cool")
 
#output
#setting is name_ind_db
#cool

The call to db_setting returns a decorator, which calculates the index of argname in the list of arguments of the function to be decorated. The rest works just like the previous example, if the argument was not passed as a positional or named argument, it’s just added to the named arguments list. When defining the test function we don’t even need to provide a default value in this case.  This was just a silly example, but I think you get the idea.

Cheers