GitLocker: The Coding Marketplace

Description:

horastrereleased 0.4.3

Human-oriented abstract syntax tree (AST) parser/unparser for Python 3 that doesn’t discard comments.

This package provides new AST node types (Comment and Directive) which inherit from nodes in typed_ast.ast3 module.
Additionally, it provides implementation of parser and unparser for the extended AST allowing
straightforward readable code generation.
Simple example of how to use this package:
from horast import parse, unparse

tree = parse("""a = 1 # a equals one after this""")
print(unparse(tree))
# this will print the code with original comment
More examples in examples.ipynb.

technical details
Parser is based on built-in tokenize module, as well as community packages asttokens and typed_ast.
Unparser is essentially an extension of Unparser class from static_typing package.
Nodes provided and handled by horast are listed below.

Comment
Full line as well as end-of-line comments are parsed/unparsed correctly when they are outside
of multi-line expressions.
Currently, handling of comments within multi-line expressions is implemented only partially.

BlockComment
Not implemented yet.
This node type is intended to store consecutive full-line comments in a
single AST node, therefore simplifying handling of large blocks of comments.

Directive
Since Python doesn’t offer a direct way to convey directives in code,
comments beginning with any of the following prefixes will not be classified
as typical comments, but as directives:

if
else
endif
def
undef
ifdef
ifndef

However in Python code they still remain as usual comments.
For example, the comments in the following code will all be classified as directives.
#ifdef DEBUG
debugging = True
#else
debugging = False
#fi
However, when executed as Python, debugging will always end up False
because directives are preserved as usual comments in Python and therefore
they are ignored.
Therefore, the Directive node is not meant to enable preprocessing of
Python, at least for now.
Note: the prefix is checked exactly. See the following example:
#if something
# if some other thing
The comment in the first line will become Directive object, while the one
on the second like will become Comment object.
Currently, this node is meant to work towards AST compatibility between
Python and other languages, to aide code generation from Python AST into code
in other languages – one such use case is
*transpyle* project.

Pragma
Pragma nodes follow the same general rules as Directive nodes, i.e. they are
stored in Python code as comments, but a comment will be classified as a pragma
when it’s prefixed with a predefined prefix:

" pragma:"

Additionally, two subclasses of Pragma are defined in horast, each with its
own prefix that builds upon the generic pragma prefix:

OpenMpPragma class defines prefix " pragma: omp" and stores OpenMP pragmas.
OpenAccPragma class defines prefix " pragma: acc" and stores OpenACC pragmas.

A code snippet below contains all 3 pragma types.
# pragma: once
use_openmp = True
use_openacc = True
...
a, b = np.ndarray(...)
c = np.zeros(...)
# pragma: acc parallel copyin(a,b) copyout(c)
# pragma: acc loop gang
for y in range(ymax): # type: np.int32
# pragma: acc loop worker
for i in range(imax): # type: np.int32
# pragma: acc loop vector reduction(+: c[y][x])
for x in range(xmax): # type: np.int32
c[y, x] += a[y, i] * b[i, x]
# pragma: acc end parallel
...
# pragma: omp parallel do
for i in range(input_data.size): # type: int
# here we compute spam spam spam
heavy_compute(input_data[i])
...
And thus, in the example above:

all comments starting with " pragma: omp" become OpenMpPragma objects,
all comments starting with " pragma: acc" become OpenAccPragma objects,
all other comments starting with " pragma:" become Pragma objects,
type comments are ignored, and
all other comments become Comment objects.

Additionally, horast module provides an extensible infrastructure to define
custom Pragma subclasses, enabling user to define their own pragmas for
experimentation. The provided OpenMP and OpenACC pragma definitions serve
as examples of how to use this feature.

Include
Similarly to how pragmas are handled, if a comment begins with " include:"
prefix, it will be classified as a special kind of include directive.
Again, this will be preserved as comment in Python code, but it’s useful
for enhancing syntactic compatibility between Python and other, especially
statically compiled languages.

requirements
CPython 3.5 or later.
Python libraries as specified in requirements.txt.
Building and running tests additionally requires packages listed in test_requirements.txt.