botoplier 0.0.1rc2

Description:

botoplier 0.0.1rc2

botoplier 🚀

Enhance your AWS API interactions using botoplier, a Python library built on top
of boto3 and botocore, facilitating simplified API
operations through automated fetch, session management, and data handling features.
Features

Response un-nesting: Specify the nested attribute you're interested in, and let botoplier pull it from the results
for you.
Seamless pagination: Pagination is handled transparently - all results will be fetched when pagination is
required.
Multi-session management: Convenient management of multiple AWS sessions.
Asyncio support: Can be used synchronously or asynchronously with asyncio [1].
Simple It's not complex!


Way under 1K LOC. You can read it all in an hour.



[1] : botocore still does not support asynchronous I/O per-se, so the concurrency support relies on asyncio thread pools
at this time.
Installation
Botoplier is distributed on PyPI, and can be installed
with pip, poetry, pdm, or any other Python package manager:
pip install botoplier

smart_query
Basic examples
from botoplier.sync.smart_query import smart_query

# Describe all EC2 instances across multiple sessions and regions
result = smart_query("ec2", "describe_instances")

NB: The API names are kebab-cased while the operation names are snake_cased, which is in line with boto3.
Result shape and un-nesting
The smart_query function automatically processes and simplifies the response structure from AWS services. It neatly
organizes deeply nested data into a more readable and usable format.
Pagination detection
smart_query automatically detects and handles API pagination, ensuring that all data across pages is fetched and
returned as a single cohesive response.
Multi-session / Parallelism
botoplier was initially developed not just to reduce boilerplate around boto3 calls, but also to simplify querying
several accounts at once.
Creating / Configuring multi-session
You can create and configure multi-sessions to simultaneously work across different AWS accounts and regions:
from botoplier.sync.sessions import make_sessions

sessions = make_sessions({"account1": "123456789012"}, ["us-west-2", "us-east-1"], {"account1": "roleName"})

The above assumes that the "ambient" AWS account is able to assume the provided roles in the matching accounts.
make_sessions produces a dictionary keyed by "session key" strings, for each of which a DecoratedSession is
attached.
The sessions are cached on-disk for as long as the credentials are valid.
Bring your own sessions
DecoratedSession
A DecoratedSession is essentially a regular botocore session, with a smidge of extra information that allows the
recompute "session keys" from it. A "session key" is a unique string identifier for the session, which is somewhat human
readable, and machine parsable, like "eu-west-3-prd".
Using your own sessions
It's quite possible your AWS authentication setup is more complicated than ours, and that our default make_sessions()
helper is inappropriate as a result. You can bring your own sessions by passing
any dict[SessionKey, DecoratedSession].
from botoplier.types import DecoratedSession

# the botocore_session argument is key here
# NB: DecoratedSession inherits from boto3.Session, so you may use any of its init parameters
ds = DecoratedSession(botocore_session=..., region_name="eu-west-3")
ds.set_arn_from_sts() # You can also do this manually using .set_arn()

Parallel calls with gather_dict
Use gather_dict to await a dictionary of future results. It works nicely with session
dictionaries such as the ones provided by make_sessions like so:
from botoplier.asyncio import smart_query, make_sessions, gather_dict


async def in_async():
sessions = await make_sessions({"account1": "123456789012"}, ["us-west-2", "us-east-1"], {"account1": "roleName"})
results = await gather_dict({
sk: smart_query("s3", "list_buckets", session=session)
for sk, session in sessions.items()
})

smart_query, make_sessions and gather_dict is the botoplier trifecta. There is not much more to it - it's just
boto3, but nicer.
Environment variables
Changing the environment variable prefix
If you're writing a tool that uses botoplier, it is sometimes more expedient to
change the recognized prefix for environment variables, which defaults to BOTOPLIER.
In your src/__init__.py, or before any botoplier import:
import botoplier.config

botoplier.config.PREFIX = "MYTOOL"

CLI Usage
Botoplier ships with a simple CLI to easily inspect the output shape of AWS APIs.
Discovering default output shape
$ AWS_DEFAULT_REGION=eu-west-3 botoplier output-shape ec2 describe_host_reservations

DescribeHostReservationsResult (structure)
└── HostReservationSet: HostReservationSet (list)
└── HostReservation (structure)
├── Count: Integer (integer)
├── CurrencyCode: CurrencyCodeValues (string)
├── Duration: Integer (integer)
├── End: DateTime (timestamp)
├── HostIdSet: ResponseHostIdSet (list)
│ └── String (string)
├── HostReservationId: HostReservationId (string)
├── HourlyPrice: String (string)
├── InstanceFamily: String (string)
├── OfferingId: OfferingId (string)
├── PaymentOption: PaymentOption (string)
├── Start: DateTime (timestamp)
├── State: ReservationState (string)
├── Tags: TagList (list)
│ └── Tag (structure)
│ ├── Key: String (string)
│ └── Value: String (string)
└── UpfrontPrice: String (string)

Tree size: 21
First branching node: .HostReservationSet[]

The output here indicates the default un-nesting smart_query will adopt when calling this API. This expression means a
list of HostReservation dicts will be returned, avoiding the need to manually read through the HostReservationSet
field present at the top level of the response.
Discovering the shape with a hint
Doing a similar invocation with an extra argument explores the shape when giving a different unnest= parameter.
$ AWS_DEFAULT_REGION=eu-west-3 botoplier output-shape ec2 describe_host_reservations Tags

DescribeHostReservationsResult (structure)
└── HostReservationSet: HostReservationSet (list)
└── HostReservation (structure)
├── Count: Integer (integer)
├── CurrencyCode: CurrencyCodeValues (string)
├── Duration: Integer (integer)
├── End: DateTime (timestamp)
├── HostIdSet: ResponseHostIdSet (list)
│ └── String (string)
├── HostReservationId: HostReservationId (string)
├── HourlyPrice: String (string)
├── InstanceFamily: String (string)
├── OfferingId: OfferingId (string)
├── PaymentOption: PaymentOption (string)
├── Start: DateTime (timestamp)
├── State: ReservationState (string)
├── Tags: TagList (list)
│ └── Tag (structure)
│ ├── Key: String (string)
│ └── Value: String (string)
└── UpfrontPrice: String (string)

Tree size: 21
First branching node: .HostReservationSet[]
Key Tags lookup path: .HostReservationSet[].Tags

Note the Key Tags lookup path at the end. One can understand that by passing unnest="Tags" to smart_query, one
will receive a list of list of tag dicts as a result.
Contributing
We value contributions from the community! Please read our Contributing Guidelines and
our Code of Conduct for information on how to get involved.
Quick setup guide
We use mise and PDM for local development.
With these installed:
mise install
pdm install
pdm run botoplier --help # Test the installation of the CLI
pdm build # Build the package wheel and sdist
pdm run pytest # Run the tests
# etc...

We use pre-commit hooks to ensure code quality.
Supported versions
We currently support Python 3.9, 3.10, 3.11 and 3.12.
We may drop the support for a Python version before its end of life, to keep the codebase up to date with the latest
Python features: i.e.: we will endeavor to support either the last 3 or 4 stable Python releases.
We don't plan to support earlier versions or different runtimes, but contributions are welcome.
Roadmap
Syntax and advance type-hinting tricks are areas we'd like to improve.
The multi-region / multi-account support takes some of the tedium away. We're open
to improving this aspect further should an interested party come up with something.
Outside of this, we're happy with the limited scope of this library and are unlikely to
accept contributions widening its scope.
TODO

Replace jq by jmespath for un-nesting
Rename smart_query module to query module, to avoid confusion with smart_query function
Proper online documentation, with published object inventory