This post originated from an RSS feed registered with Python Buzz by Sidnei da Silva. Original Post: memcached Cache Manager for Zope 2 Feed Title: awkly.org Feed URL: http://awkly.org/categories.rdf?category=Python Feed Description: dreamcatching :: making your dreams come true Latest Python Buzz Posts Latest Python Buzz Posts by Sidnei da Silva Latest Posts From awkly.org

While staring at the code for the memcached python wrapper, I wondered if there was anything that could be done to make it faster, but sadly anything that I tried just made it slower.

One of the things I tried was to read and write directly to the socket instead of using loads and dumps from the cPickle module. The win was unnoticeable for small objects, and sadly it seemed to be slower for large objects.

Another thing that I tried was to optimize the recv method of the _Host class, which looks like this:

def recv(self, rlen):
  buf = ''
  recv = self.socket.recv
  while len(buf) < rlen:
      buf = buf + recv(rlen - len(buf))
  return buf

I've tried five or six variations that would do the same, and yet they were still slower than this one. I'm particularly suprised that:

buf = buf + recv(rlen - len(buf))

seems to be faster than:

buf += recv(rlen - len(buf))

And that the two calls to len(buf) when replaced by a single call also make the code slower. My bet is that's because recv() always finds a filled buffer so the loop only happens once.

I'm sure the fine folks at the Python Brasil list would love to discuss these results.

All in all, I've only got some benchmarking done and not much anything else. Here's some results from the benchmark. Each result is for 10 iterations (using the timeit module). The object stored in memcached is an instance of a simple class with a string attribute, where the string length is what is indicated here as Size:

Benchmarking...
Size: 0
time for set() 0.00602293014526
time for get() 0.00676512718201
Size: 131072
time for set() 0.441082000732
time for get() 0.0506250858307
Size: 262144
time for set() 0.571678161621
time for get() 0.138868808746
Size: 393216
time for set() 0.657498121262
time for get() 0.221096038818
Size: 524288
time for set() 0.702223062515
time for get() 0.381999015808

For completeness, this is the code used for the benchmark:

if __name__ == "__main__":
    import timeit
    servers = ["127.0.0.1:11211"]
    mc = Client(servers, debug=1)
    class FooStruct:
        def __init__(self, value="baz"):
            self.bar = value
    print "Benchmarking..."

    for i in xrange(0, (1<<19)+1, 1<<17):
        print 'Size: %d' % i
        f = FooStruct('x' * i)

        t = timeit.Timer("mc.set('foo_timer', f)",
                         'from __main__ import mc, f')
        print 'time for set()', t.timeit(10)

        t = timeit.Timer("mc.get('foo_timer')",
                         'from __main__ import mc, f')
        print 'time for get()', t.timeit(10)