GitLocker: The Coding Marketplace

Description:

libmc 1.4.6

libmc is a memcached client library for Python without any other
dependencies at runtime. It’s mainly written in C++ and Cython and
can be considered a drop in replacement for libmemcached and
python-libmemcached.
libmc is developed and maintained by Douban Inc. Currently, it is
working in a production environment, powering all web traffic on
douban.com
(english wiki).

Build and Installation
For users:
pip install libmc
Usage:
import libmc

mc = libmc.Client(['localhost:11211', 'localhost:11212'])
mc.set('foo', 'bar')
assert mc.get('foo') == 'bar'

Under the hood
Under the hood, libmc consists of 2 parts: an internal, fully-functional
memcached client implementation in C++ and a Cython wrapper around that
implementation. Dynamic memory allocation and memory-copy are slow, so
we’ve tried our best to avoid them. libmc also supports the set_multi
command, which is not natively supported by the memcached
protocol.
Some techniques have been applied to make set_multi command extremely fast
in libmc (compared to similiar libraries).

Configuration
import libmc
from libmc import (
MC_HASH_MD5, MC_POLL_TIMEOUT, MC_CONNECT_TIMEOUT, MC_RETRY_TIMEOUT
)

mc = libmc.Client(
[
'localhost:11211',
'localhost:11212',
'remote_host',
'remote_host mc.mike',
'remote_host:11213 mc.oscar'
],
do_split=True,
comp_threshold=0,
noreply=False,
prefix=None,
hash_fn=MC_HASH_MD5,
failover=False
)

mc.config(MC_POLL_TIMEOUT, 100) # 100 ms
mc.config(MC_CONNECT_TIMEOUT, 300) # 300 ms
mc.config(MC_RETRY_TIMEOUT, 5) # 5 s

servers: a list of memcached server addresses. Each address
should be formated as hostname[:port] [alias], where port and
alias are optional. If port is not given, the default port 11211
will be used. If given, alias will be used to compute the server hash,
which would otherwise be computed based on host and port
(i.e. whichever portion is given).
do_split: splits large values (up to 10MB) into chunks (<1MB). The
memcached server implementation will not store items larger than 1MB,
however in some environments it is beneficial to shard up to 10MB of data.
Attempts to store more than that are ignored. Default: True.
comp_threshold: compresses large values using zlib. If
buffer length > comp_threshold > 0 (in bytes), the buffer will be
compressed. If comp_threshold == 0, the string buffer will never be
compressed. Default: 0
noreply: controls memcached’s
noreply feature.
Default: False
prefix: The key prefix. default: ''
hash_fn: hashing function for keys. possible values:

MC_HASH_MD5
MC_HASH_FNV1_32
MC_HASH_FNV1A_32
MC_HASH_CRC_32

default: MC_HASH_MD5
NOTE: fnv1_32, fnv1a_32, crc_32 implementations in libmc are
per each spec, but they’re not compatible with corresponding
implementions in libmemcached.

failover: Whether to failover to next server when current server
is not available. Default: False
MC_POLL_TIMEOUT Timeout parameter used during set/get procedure.
Default: 300 ms
MC_CONNECT_TIMEOUT Timeout parameter used when connecting to
memcached server in the initial phase. Default: 100 ms
MC_RETRY_TIMEOUT When a server is not available due to server-end
error, libmc will try to establish the broken connection in every
MC_RETRY_TIMEOUT s until the connection is back to live. Default:
5 s

NOTE: The hashing algorithm for host mapping on continuum is always
md5.

Contributing to libmc
Feel free to send a Pull Request. For feature requests or any
questions, please open an Issue.
For SECURITY DISCLOSURE, please disclose the information responsibly
by sending an email to security@douban.com directly instead of creating
a GitHub issue.

FAQ

Does libmc support PHP?
No, but, if you like, you can write a wrapper for PHP based on the C++
implementation.

Is Memcached binary protocol supported ?
No. Only Memcached ASCII protocol is supported currently.

Why reinventing the wheel?
Before libmc, we were using
python-libmemcached,
which is a python extention for
libmemcached.
libmemcached is quite weird and buggy. After nearly one decade, there’re
still some unsolved bugs.

Is libmc thread-safe ?
Yes. libmc.ThreadedClient is a thread-safe client implementation. To hold
access for more than one request, libmc.ClientPool can be used with Python
with statements. libmc.Client, however, is a single-threaded memcached
client. If you initialize a standard client in one thread but reuse that in
another thread, a Python ThreadUnsafe Exception will be raised.

Is libmc compatible with gevent?
Yes, with the help of greenify,
libmc is friendly to gevent. Read tests/shabby/gevent_issue.py for
details. libmc.ThreadedClient and libmc.ClientPool are not compatible.
[1]
Notice:
gevent.monkey.patch_all() will override
threading.current_thread().ident to Greenlet’s ID,
this will cause libmc to throw a ThreadUnSafe error
or run into dead lock, you should only patch the things
that you need, e.g.
from gevent import monkey
monkey.patch_socket()

Acknowledgments

Thanks to @fahrenheit2539 and
the llvm project for the standalone.
SmallVector
implementation.
Thanks to @miloyip for the high
performance i64toa
implementation.
Thanks to Ivan Novikov for the
research in THE NEW PAGE OF INJECTIONS BOOK: MEMCACHED
INJECTIONS.
Thanks to the PolarSSL project for the md5 implementation.
Thanks to @lericson for the benchmark
script in
pylibmc.
Thanks to the libmemcached project and some other projects possibly
not mentioned here.

Contributors

@mckelvin
@zzl0
@windreamer
@lembacon
@seansay
@mosasiru
@jumpeiMano

Who is using

豆瓣
下厨房
Some other projects on GitHub
Want to add your company/organization name here?
Please feel free to send a PR!

Documentation
https://github.com/douban/libmc/wiki

Footnotes

[1]
In order to use a single executable for multiple greenlet contexts,
gevent has to copy thread memory
to and from the same stack space. This doesn’t affect Python references,
which are handed off through gevent, but makes it impossible for shared
libraries to pass memory addresses across greenlets, which is required for
the worker pool.

LICENSE
Copyright (c) 2014-2020, Douban Inc. All rights reserved.
Licensed under a BSD license:
https://github.com/douban/libmc/blob/master/LICENSE.txt