aioredis-cluster 2.7.0

Description:

aiorediscluster 2.7.0

aioredis_cluster

Redis Cluster support for aioredis (support only v1.x.x).
Many implementation features were inspired by go-redis project.
Requirements

Python 3.8+
async_timeout (only for Python < 3.11)

Features

commands execute failover (retry command on other node in cluster)
support resharding replies ASK/MOVED
restore cluster state from alive nodes
one node is enough to know the topology and initialize client
cluster state auto reload

Limitations
Commands with limitations

Keys in mget/mset must provide one key slot.
# works
await redis.mget("key1:{foo}", "key2:{foo}")

# throw RedisClusterError
await redis.mget("key1", "key2")



Commands are not supported
Redis methods below do not works and not supported in cluster mode implementation.
cluster_add_slots
cluster_count_failure_reports
cluster_count_key_in_slots
cluster_del_slots
cluster_failover
cluster_forget
cluster_get_keys_in_slots
cluster_meet
cluster_replicate
cluster_reset
cluster_save_config
cluster_set_config_epoch
cluster_setslot
cluster_readonly
cluster_readwrite
client_setname
shutdown
slaveof
script_kill
move
select
flushall
flushdb
script_load
script_flush
script_exists
scan
iscan
quit
swapdb
migrate
migrate_keys
wait
bgrewriteaof
bgsave
config_rewrite
config_set
config_resetstat
save
sync
pipeline
multi_exec

But you can always execute command you need on concrete node on cluster with usual aioredis.RedisConnection, aioredis.ConnectionsPool or high-level aioredis.Redis interfaces.
Installation
pip install aioredis-cluster

Usage
import aioredis_cluster

redis = await aioredis_cluster.create_redis_cluster([
"redis://redis-cluster-node1",
])

# or
redis = await aioredis_cluster.create_redis_cluster([
"redis://redis-cluster-node1",
"redis://redis-cluster-node2",
"redis://redis-cluster-node3",
])

# or
redis = await aioredis_cluster.create_redis_cluster([
("redis-cluster-node1", 6379),
])

await redis.set("key", "value", expire=180)

redis.close()
await redis.wait_closed()

License
The aioredis_cluster is offered under MIT license.
Changes
2.7.0 (2023-12-18)
Rework PubSub and fix race conditions (#27)

add aioredis_cluster.aioredis.stream module
rework PubSub command execution flow for prevent race conditions on spontaneously server channels unsubscribe push
make fully dedicated RedisConnection implementation for cluster
RedisConnection once entered in PubSub mode was never exit in them, because is too hard handle spontaneously unsubscribe events from Redis with simultaneously (P|S)UNSUBSCRIBE manually calls
fully rewrite handling PUB/SUB replies/events
for Cluster, RedisConnection and ConnectionsPool in_pubsub indicates flag when connector have in pubsub mode connections instead number of PUB/SUB channels
add key slot handling for sharded PubSub channels in non-cluster dedicate RedisConnection
fix and improve legacy aioredis tests
improve support for py3.12
improve support for Redis 7.2

2.6.0 (2023-11-02)

fix stuck aioredis.Connection socket reader routine for sharded PUB/SUB when cluster reshard and Redis starts respond MOVED error on SSUBSCRIBE commands #24

2.5.0 (2023-04-03)

improve connection creation timeout
do not lose connection in Pool while execute PING probe
respect Pool.minsize in idle connections detector
shuffle startup nodes for obtain cluster state

2.4.0 (2023-03-08)

add support Sharded PUB/SUB
new methods and properties spublish, ssubscribe, sunsubscribe, pubsub_shardchannels, pubsub_shardnumsub, sharded_pubsub_channels
drop support Python 3.6, 3.7
add support Python 3.11
idle connections detection in connections pool
change acquire connection behaviour from connection pool. Now connection acquire and release to pool by LIFO way for better idle connection detection
deprecated state_reload_frequency option from create_cluster factory was removed

2.3.1 (2022-07-29)

fix bypass username argument for pool creation

2.3.0 (2022-07-26)

add support Redis 6 AUTH command with username
factories create_cluster, create_redis_cluster, aioredis_cluster.aioredis.create_connection now support username argument
add auth_with_username method for AbcConnection, AbcPool and impementations

2.2.2 (2022-07-19)

fix problem when RedisConnection was GC collected after unhandled asyncio.CancelledError
fix default db argument for pool/connection in cluster mode

2.2.1 (2022-07-18)

(revert) apply cluster state only if cluster metadata is changed

2.2.0 (2022-07-18)

fetch several cluster state candidates from cluster for choose best metadata for final local state
apply cluster state only if cluster metadata is changed
FIX: handle closed pubsub connection before gc its collected that trigger Task was destroyed but it is pending! message in log
improve logging in state loader

2.1.0 (2022-07-10)

fix bug when ConnectionsPool.acquire() is stuck because closed PUB/SUB connection is not cleanup from used set
fix ConnectionsPool.acquire() incorrect wakeup order for connection waiters when connection is released
ConnectionsPool.execute() now acquire dedicate connection for execution if command is blocking, ex. BLPOP
ConnectionsPool.execute() now raises ValueError exception for PUB/SUB family command
In ConnectionsPool PUB/SUB dedicated connections now is closing on close() call
add aioredis_cluster.abc.AbcConnection abstract class
add property readonly and method set_readonly() for aioredis_cluster.abc.AbcConnection and aioredis_cluster.abc.AbcPool
aioredis_cluster.Cluster now require pool_cls implementation from aioredis_cluster.abc.AbcPool
add ssl argument for factories create_cluster, create_redis_cluster and Cluster constructor
add 10% jitter for cluster state auto reload interval
fix incorrect iterate free connections in select(), auth() methods for ConnectionsPool

2.0.0 (2022-06-20)

include aioredis==1.3.1 source code into aioredis_cluster._aioredis and introduce aioredis_cluster.aioredis but for compatible and migration period
this release have not backward incompatible changes
DEPRECATION WARNING: you must migrate from import aioredis to import aioredis_cluster.aioredis because aioredis_cluster starts vendorize aioredis package and maintain it separately. Using aioredis package will be removed in v3
fix reacquire connection in aioredic.ConnectionsPool after Redis node failure

1.8.0 (2022-05-20)

Add xadd_620 commands method for support XADD options for Redis 6.2+

1.7.1 (2021-12-15)

add ClusterState.slots_assigned
require reload cluster state for some cases with UncoveredSlotError

1.7.0 (2021-12-15)

add execute_timeout for Manager
improve cluster state reload logging
reduce number of addresses to fetch cluster state
acquire dedicate connection from pool to fetch cluster state
extend ClusterState by new attributes: state, state_from, current_epoch

1.6.1 (2021-11-23)

fix keys extraction for XREAD and XREADGROUP commands

1.6.0 (2021-11-20)

make public Address, ClusterNode and ClusterState structs. Available by import from aioredis_cluster import
Cluster provides some new helpful methods:

get_master_node_by_keys(*keys) - return master ClusterNode which contains keys keys
create_pool_by_addr(addr, **kwargs) - create connection pool by addr and return pool wrapped by commands_factory from Cluster constructor. By default is aioredis_cluster.RedisCluster instance.
get_cluster_state() - return ClusterState instance with recent known cluster state received from Redis cluster
extract_keys(command_sequence) - returns keys of command sequence


drop pytest-aiohttp plugin for tests
add pytest-asyncio dependency for tests
switch asynctest -> mock library for aio tests
drop attrs dependency. For Python 3.6 you need install dataclasses
fix extract keys for BLPOP/BRPOP commands
add support keys extraction for ZUNION, ZINTER, ZDIFF, ZUNIONSTORE, ZINTERSTORE, ZDIFFSTORE commands
acquire dedicate connection from pool for potential blocking commands like BLPOP, BRPOP, BRPOPLPUSH, BLMOVE, BLMPOP, BZPOPMIN, BZPOPMAX, XREAD, XREADGROUP

1.5.2 (2020-12-14)

README update

1.5.1 (2020-12-11)

speedup crc16. Use implementation from python stdlib

1.5.0 (2020-12-10)

remove state_reload_frequency from ClusterManager. state_reload_interval now is one relevant option for state auto reload
default state_reload_interval increased and now is 300 seconds (5 minutes)
commands registry loads only once, on cluster state initialize
improve failover. First connection problem cause retry to random slot replica
improve python3.9 support
default idle_connection_timeout now is 10 minutes

1.4.0 (2020-09-08)

fix aioredis.locks.Lock issue (https://github.com/aio-libs/aioredis/pull/802, bpo32734)
now aioredis_cluster.Cluster do not acquire dedicate connection for every execute
aioredis_cluster now requires python>=3.6.5

1.3.0 (2019-10-23)

improve compatible with Python 3.8
improve failover logic while command timed out
read-only commands now retries if attempt_timeout is reached
add required dependeny async_timeout
aioredis dependency bound now is aioredis >=1.1.0, <2.0.0

1.2.0 (2019-09-10)

add timeout for command execution (per execution try)
add Cluster option attempt_timeout for configure command execution timeout, default timeout is 5 seconds
Cluster.execute_pubsub() fixes

1.1.1 (2019-06-07)

CHANGES fix

1.1.0 (2019-06-06)

Cluster state auto reload
new state_reload_frequency option to configure state reload frequency
new state_reload_interval option to configure state auto reload interval
follow_cluster option enable load cluster state from previous cluster state nodes
establish connection only for master nodes after cluster state load
change default commands_factory to aioredis_cluster.RedisCluster instead aioredis.Redis
all cluster info commands always returns structs with str, not bytes
keys_master and all_masters methods now try to ensure cluster state instead simply raise exception if connection lost to cluster node, for example
max_attempts always defaults fix

1.0.0 (2019-05-29)

Library full rewrite
Cluster state auto reload
Command failover if cluster node is down or key slot resharded

0.2.0 (2018-12-27)

Pipeline and MULTI/EXEC cluster implementation with keys distribution limitation (because cluster)

0.1.1 (2018-12-26)

Python 3.6+ only

0.1.0 (2018-12-24)

Initial release based on aioredis PR (https://github.com/aio-libs/aioredis/pull/119)

Contributors

Anton Ilyushenkov
Vadim Pushtaev
erastov
roman901
Alexander Malev