radula 0.8.7

Description:

radula 0.8.7

radula======radula is a small utility to add some friendliness to`RadosGW <http://ceph.com/docs/master/man/8/radosgw/>`__ for our teamworking with `Ceph <http://ceph.com/>`__ for S3-like storage. Littlemore than a wrapper for`boto <http://boto.readthedocs.org/en/latest/>`__ radula saves us timeand headache by using nice defaults.The primary functions for the current version are- Inspect radosgw bucket/key ACLs- Spot differences between bucket and key ACLs- Allow or disallow user read/write to buckets and keys- When modifying a bucket ACL, modify the ACLs for keys as well- Verify uploads using checksums- Upload using multiple threadsThe name, "radula", is a `cephalopod-relatedterm <https://en.wikipedia.org/wiki/Radula#In_cephalopods>`__ that hitclose to RADOS. It's not a tongue, or really even teeth; it's more likeif your tongue had teeth. Spooky.Installation------------Install radula from `pypi <https://pypi.python.org/pypi>`__ using``pip``.:: pip install radulaThe ``radula`` command should be available in your ``PATH‘‘.Testing−−−−−−−Installthepippackageslistedin‘‘testing−requirements.txt‘‘andrun‘‘nosetests‘‘.:: pip install -U -r testing-requirements.txt Missing open brace for subscriptMissing open brace for subscript usage: radula [-h] [--version] [-r] [-w] [-t THREADS] [-p PROFILE] [-d DESTINATION] [-f] [-y] [-c CHUNK_SIZE] [-l] [-L LOG_LEVEL] [-v] [-n] [-z] [-e] [-A] [-i] [-k] [--no-acl] [{acls,get-acl,set-acl,compare-acl,sync-acl,allow,allow-user,disallow,disallow-user,mb,make-bucket,rb,remove-bucket,lb,list-buckets,put, up,upload,get,dl,download,mpl,mp-list,multipart-list,mpc,mp-clean,multipart-clean,rm,remove,keys,ls,list,info,size,etag,remote-md5,remote-rehash,verif y,sc,streaming-copy,copy,cat,url,get-url,local-md5,profiles}] [subject] [target] ... RadosGW client positional arguments: {acls,get-acl,set-acl,compare-acl,sync-acl,allow,allow-user,disallow,disallow-user,mb,make-bucket,rb,remove-bucket,lb,list-buckets,put,up,upload, get,dl,download,mpl,mp-list,multipart-list,mpc,mp-clean,multipart-clean,rm,remove,keys,ls,list,info,size,etag,remote-md5,remote-rehash,verify,sc, streaming-copy,copy,cat,url,get-url,local-md5,profiles} command subject Subject target Target remainder Additional targets for supporting commands. See README optional arguments: -h, --help show this help message and exit --version Prints version number -r, --read During a user grant, permission includes reads -w, --write During a user grant, permission includes writes -t THREADS, --threads THREADS Number of threads to use for uploads. Default=3 -p PROFILE, --profile PROFILE Boto profile. Overrides AWS_PROFILE environment var -d DESTINATION, --destination DESTINATION Destination boto profile, required for streaming copy -f, --force Overwrite local files without confirmation -y, --verify Verify uploads after they complete. Uses --threads. When passed a destination profile, download and hash keys on both ends -c CHUNK_SIZE, --chunk CHUNK_SIZE multipart upload chunk size in bytes. -l, --long-keys prepends bucketname to key results. -L LOG_LEVEL, --log-level LOG_LEVEL Log level, [DEBUG, 10, INFO, 20, etc] -v, --verbose Verbose. Equiv to -L DEBUG -n, --dry-run Print would-be deletions without deleting -z, --resume Resume uploads if needed. -e, --encrypt Store content encrypted at rest -A, --all-buckets act upon all buckets (info only) -i, --ignore-existing Calmly skip existing files; an opposite -f (otherwise errors) -k, --preserve-key When downloading, preserve paths in keys --no-acl When uploading, do not sync key ACL with the bucket ACL. Normally would.Examples--------This is a quick walkthrough of the features so far. In these scenarios,we acting as the user ``bibby``, who owns the rados bucket ``mybucket``.In some of the examples, we'll be manipulating the access to this bucketfor a second user called ``fred``.Contained in the bucket are two regular files: ``hello`` and ``world``.List available profiles~~~~~~~~~~~~~~~~~~~~~~~See `Boto docs <http://boto.cloudhackers.com/en/latest/boto_config_tut.html#credentials>`__ for working with profiles.:: [bibby@machine ~]radulaprofilesherethere∗DEFAULTDisplayingbucketACL                     ::[bibby@machine ] radula get-acl mybucket ACL for bucket: mybucket [CanonicalUser:OWNER] Andrew Bibby = FULL_CONTROLThe command ``get-acl`` prints the acl. radula assumed that the term``mybucket`` was a bucket, being that it was a lone term.Displaying key ACL~~~~~~~~~~~~~~~~~~:: [bibby@machine ~]radulaget−aclmybucket/helloACLforkey:mybucket/hello[CanonicalUser:OWNER]AndrewBibby=FULLCONTROLBecausethetermcontainedaslash,thesubjectiscorrectlyidentifiedas‘‘hello‘‘withinthebucket‘‘mybucket‘‘.ComparingACLs−Keysinbucket                               ::[bibby@machine ] radula compare-acl mybucket Bucket ACL for: mybucket [CanonicalUser:OWNER] Andrew Bibby = FULL_CONTROL --------- Keys with identical ACL: 2 Keys with different ACL: 0The ``compare-acl`` command on a bucket will report of the *sameness* ofACLs across the keys as compared to the bucket. We'll see this againlater in another example.This *can* be run against one key, limiting the compared objects to theone key against its bucket:: [bibby@machine ~]radulacheck−aclmybucket/helloBucketACLfor:mybucket[CanonicalUser:OWNER]AndrewBibby=FULLCONTROL−−−−−−−−−KeyswithidenticalACL:1KeyswithdifferentACL:0SetacannedACL                CansettheACLofabucketorkeytooneofthefourAWS"canned"policiesusing‘‘set−acl‘‘.Inthisscenario,the∗subject∗canbeabucketorakey,withthe∗target∗beingacannedpolicyname.::[bibby@machine ] radula set-acl mybucket/hello public-read << prints the output of get-acl after completing the operationChanging the ACL on a bucket **will** will be applied to the keys aswell, potentially overwriting any custom access given to keys. Run``compare-acl`` before setting the bucket ACL to discover any specialdifferences, as they may need to be recreated after the ``set-acl``operation completes.Sync ACLs~~~~~~~~~Should a difference of ACL had appeared, we could forcefully replace allkey ACLs with the bucket's ACL using ``sync-acl``.:: [bibby@machine ~]radulasync−aclmybucketBucketACLfor:mybucket[CanonicalUser:OWNER]AndrewBibby=FULLCONTROL−−−−−−−−−Settingbucket′sACLonhelloSettingbucket′sACLonworldThisisa‘‘PUT‘‘command,soitdoesn′tbothertolookatthecurrentACLforthekeys;itjustputsacopyofthebucket′sownACL.‘‘sync−acl‘‘canbedoneonasinglekeyaswell.::[bibby@machine ] radula sync-acl mybucket/world Setting bucket's ACL on worldGranting access to a key~~~~~~~~~~~~~~~~~~~~~~~~To grant access to another user, we'll make use of some new flags.``-r`` and/or ``-w`` to indicate read and write. A grant may have one orboth of ``rw``. If both are absent, ``read`` is assumed. Permissions areseparate, so it is possible to have a *write-only* grant.For permission grants the *subject* is the **user** (as far as the usageformat in the help text goes), and the *target* is the **key orbucket**.:: [bibby@machine ~]radulaallowfredmybucket/hellograntingREADtofredonkeyhelloMultiplegrantstothesameuserforthesamepermissionarepossibleinradosandons3,butradulawillguardagainstthatandignoretheduplicateentry.Here,we′lladd"read−write":::[bibby@machine ] radula -wr allow fred mybucket/hello User fred already has READ for key hello, skipping granting WRITE to fred on key helloGranting access to a bucket~~~~~~~~~~~~~~~~~~~~~~~~~~~| Granting access to a bucket works the same way.| When a bucket ACL is modified, **so are all of its keys**. That action is really the whole purpose behind radula.:: [bibby@machine ~]radula−wrallowfredmybucketgrantingREADtofredonbucketmybucketgrantingWRITEtofredonbucketmybucketUserfredalreadyhasREADforkey<Key:mybucket,hello>,skippingUserfredalreadyhasWRITEforkey<Key:mybucket,hello>,skippinggrantingREADtofredonkey<Key:mybucket,world>grantingWRITEtofredonkey<Key:mybucket,world>Withboth‘‘allow‘‘and‘‘disallow‘‘,ifanACLdifferenceexistsbetweenthebucketandakey,thatdifferencemaystillexistafterthemodification.Withthesecommands,wearen′t∗∗syncing∗∗amodifiedbucketACLdowntothekeys;we′reapplyingthesamesingularchangetoeachtargetindividually.Disallow(bucketsandkeys)                           Removingpermissionsworkssimilarlytograntingaccess,butwithsomedifferences.Oneassumptionisabouttheomissionoftheread−writeflags;Ifneitherarepresent,bothpermissionsareremoved.+−−−−−−−−−+−−−−−−−−−+−−−−−−−−−−+|start|flags|result|+=========+=========+==========+|RW|−r|W|+−−−−−−−−−+−−−−−−−−−+−−−−−−−−−−+|RW|−w|R|+−−−−−−−−−+−−−−−−−−−+−−−−−−−−−−+|RW|−rw|−|+−−−−−−−−−+−−−−−−−−−+−−−−−−−−−−+|RW|−|−|+−−−−−−−−−+−−−−−−−−−+−−−−−−−−−−+ACLsforthekeysaremodifiedfirst.Theuser′saccesscannotbetakenawayfromthebucketifitstillexistsforoneofitskeys,sothechangestakeplacefrombottomup.Creatingandifferenceandsyncingdown                                       Startingwithablankslate:::[bibby@machine ] radula -wr disallow fred mybucket No change for <Key: mybucket,hello> No change for <Key: mybucket,world> No change for mybucketGive ``fred`` read on the bucket:: [bibby@machine ~]radula−rallowfredmybucketgrantingREADtofredonbucketmybucketgrantingREADtofredonkey<Key:mybucket,hello>grantingREADtofredonkey<Key:mybucket,world>Give‘‘fred‘‘writeononekey::[bibby@machine ] radula -w allow fred mybucket/world granting WRITE to fred on key worldConfirm the difference..:: [bibby@machine ~]radulacompare−aclmybucketBucketACLfor:mybucket[CanonicalUser:OWNER]AndrewBibby=FULLCONTROL[CanonicalUser]FredFredricks=READ−−−−−−−−−Differenceinworld:[CanonicalUser:OWNER]AndrewBibby=FULLCONTROL[CanonicalUser]FredFredricks=READ[CanonicalUser]FredFredricks=WRITEKeyswithidenticalACL:1KeyswithdifferentACL:1Plowthekeyswiththebucket′ssettings.::[bibby@machine ] radula sync-acl mybucket Bucket ACL for: mybucket [CanonicalUser:OWNER] Andrew Bibby = FULL_CONTROL [CanonicalUser] Fred Fredricks = READ --------- Setting bucket's ACL on hello Setting bucket's ACL on world [bibby@machine ~]You can't use 'macro parameter character #' in math modeYou can't use 'macro parameter character #' in math mode tree projroot/ projroot/ ├── subdir_a │   ├── ef90d4f2 │   └── efd7f715 └── subdir_b ├── 10eaf5f0 ├── 80920f14 ├── a6fcadbf ├── a8dd1085 └── third_dir ├── 980a978f └── e50f86feUploading `projroot` will copy the directory structure at the location specified. *Beware: full paths (``/home/user/..``) given as sources will upload to keys using that full path.:: radulaupprojrootbucket/projects<snip> radula -p abibby keys abibby/projects/\* projects/projroot/subdir_a/ef90d4f2 projects/projroot/subdir_a/efd7f715 projects/projroot/subdir_b/10eaf5f0 projects/projroot/subdir_b/80920f14 projects/projroot/subdir_b/a6fcadbf projects/projroot/subdir_b/a8dd1085 projects/projroot/subdir_b/third_dir/980a978f projects/projroot/subdir_b/third_dir/e50f86feBecause keys are inherently flat on s3, to download recursively you'll need a combination of a glob pattern and the ``--preserve-key`` flag.:: radula−−preserve−keydlbucket/projects/projroot/Theentirekeyisusedtocreatethelocalstructure,sointhiscase,the‘‘projects‘‘dirwillberecreatedifithadgonemissing.cat   Analternativeto‘download‘is‘cat‘,whichprintsthecontentsofaremotesubjectto‘stdout‘.:: echo "Hello there you" > hello radulauphellomybucket/helloINFO:radula:Finisheduploading16.00Bin0.08s(188.82Bps) radula cat mybucket/hello Hello there youIn radula 0.7+, `cat` accept the `-c`,`--chunk-size` parameter to print part of the remote file.Unique to this command is that the chunk param can be a range of integers or humanized units.If humanized units (ie, `2kb`) are used, they'll be converted into integer to conform with the[HTTP Range header spec](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35).When using a range query, the end of the range may be omitted to include everything fromthe starting position to the end of the file.Omitting the first argument is not supported. Starting a range with zero (`0-n`) *does work*, but it is recommended to simply provide `n` by itself, because the **range in inclusive**. The range `0-100` wouldoutput 101 bytes, while input `100` returns 100.A `ValueError` will be raised if end of the range is before the starting position.:: # first two bytes You can't use 'macro parameter character #' in math modeYou can't use 'macro parameter character #' in math mode radula -c '2-' cat mybucket/hello llo # first byte to second byte (inclusive) You can't use 'macro parameter character #' in math modeYou can't use 'macro parameter character #' in math mode radula get-url bibby/foo https://s3-host/bibby/foo?Signature={signature}&Expires={expire}&AWSAccessKeyId={access_key} # many files with 15 min custom expire [bibby@machine ~]Misplaced &Misplaced & radula --dry-run rm mybucket/x DRY-RUN: rm mybucket/x [bibby@machine ~]You can't use 'macro parameter character #' in math modeYou can't use 'macro parameter character #' in math mode radula mp-list mybucket bibby ones.img 2~Q8r-pWTmMTbx_rhHa8-u3I3m-vjCF5F Andrew Bibby 2015-09-23T19:39:14.000Z bibby zeros.img 2~MvM7KTr2sMcS_SfVzWO7T0chzJRUqvm Andrew Bibby 2015-09-23T19:35:44.000Z # list multipart uploads for a key You can't use 'macro parameter character #' in math modeYou can't use 'macro parameter character #' in math mode radula mp-clean mybucket/zeros.img INFO:root:Canceling zeros.img 2~MvM7KTr2sMcS_SfVzWO7T0chzJRUqvm True # clean multipart uploads for a bucket $ radula mp-list mybucket INFO:root:Canceling ones.img 2~Q8r-pWTmMTbx_rhHa8-u3I3m-vjCF5F TrueStreaming Copy--------------Since radula 0.5.0, users are able to copy between different cephinstallations, or different buckets within the same installation,without copying to the local disk. To facilitate this in the friendliestpossible manner, we've extended the ``boto`` configuration slightly tobe able to specify a separate s3 host for a particular profile.The ``profile`` sections of ``~/.boto`` or ``/etc/boto.cfg`` can nowaccept the following items that are not supported by regular boto:- host (string)- port (int)- is\_secure (bool)An example extended profile:: [profile second_ceph] aws_access_key_id = wxyz... aws_secret_access_key = 9765432... host = second.ceph.of.mine port = 8184The commans ``streaming-copy`` and ``sc`` are equivalent. For theseexample, I've chosen to use ``sc``.When copying, the ``-p`` flag will apply the aws\_profile for the*source*/subject. Omitting this flag will use the default botocredentials for the source.The ``-d`` flag will specify the profile used for the*destination*/target to receive the files. Naming ``-d Default`` willuse the default boto credentials for the destination.Copy a file from first-ceph to second-ceph~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~``radula -d second sc mybucket/file other_bucket/file``The above command used the default boto profile to send ``file`` from``mybucket`` located on the default ceph to the ceph defined in theprofile named ``second``.Copy a file from second-ceph to first-ceph~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~``radula -p second -d Default sc other_bucket/file mybucket/file``This is the inverse of the previous example. Using the ``second``profile as the source/subject (as specified by ``-p second``), we'retransfering a file to ``mybucket/file`` located on the default s3 usingthe default profile (as specified by ``-d Default``).Copy profile to profile~~~~~~~~~~~~~~~~~~~~~~~Avoiding the use of default profiles all together, you can copy usingboth ``-p`` and ``-d`` flags.``radula -p here -d there sc here/stuff there/stuff``