python2 1.2

Description:

python2 1.2

A library for running Python 2 code from a Python 3 application.

Effortlessly harness the power and convenience of Python 2… in Python 3!


Why?
Why not??
This library was created for more whimsical than practical reasons. In theory,
it could be used to interface with legacy Python 2 code which for one reason or
another cannot be ported to Python 3.


Installation
python2 requires a working install of both Python 2 and Python 3.
Currently the library has only been tested with Python 2.7 and Python 3.4, 3.5,
and 3.6.
To install the package:
pip install -U python2
If using virtualenvs, you will need to create separate Python 2 and 3
virtualenvs, and install the package into both.


Usage
To begin working with Python 2, import the package in Python 3 and create a new
Python2 object:
>>> from python2.client import Python2
>>> py2 = Python2('/path/to/python2/executable')
This object is our gateway to the Python 2 world. Python 2 builtins can be
accessed as attributes of the Python2 object. Let’s use Python 2’s
__import__() function to import the deprecated sha module, which was
removed in Python 3:
>>> py2_sha = py2.__import__('sha')
>>> py2_sha.sha('abc')
<Py2Object <sha1 HASH object @ 0x107463c30>>
Ahh, just like the good ol’ days. You can deprecate a module but you can’t
deprecate the human spirit!
We can use the Python2.project() method to convert Python 3 objects to
Python 2:
>>> py2.project(1)
<Py2Object 1>
>>> py2.project('foo')
<Py2Object u'foo'>
You can use Python2.lift() to lift Python 2 objects back to Python 3. For
container types, use Python2.deeplift() to recursively perform the lifting.
Py2Object instances have special properties _ and __ to perform the
equivalent operations:
>>> o = py2.project([1, 2, 3])
>>> o
<Py2Object [1, 2, 3]>
>>> o._
[<Py2Object 1>, <Py2Object 2>, <Py2Object 3>]
>>> o.__
[1, 2, 3]
Python 2 objects can be used pretty much like regular Python 3 objects. You
can also freely mix and match with Python 3 builtin types:
>>> x = py2.project(1)
>>> x
<Py2Object 1>
>>> str(x)
'1'
>>> x + 1
<Py2Object 2>
>>> d = py2.dict(foo=x, bar=None)
>>> d['foo'] is x
True
>>> del d['foo']
>>> d
<Py2Object {u'bar': None}>
>>> d.__
{'bar': None}
If you just want to execute some Python 2 code directly, you can use the
Python2.exec() method. This method accepts a string containing Python 2
code and an optional dict representing the scope to execute the code in, and
returns the resulting scope after executing the code. This can be used to
define new Python 2 classes and functions:
>>> scope = py2.exec("""
... def foo(x):
... return x + 1
... """)
>>> foo = scope['foo']
>>> foo(2)
<Py2Object 3>
If an exception occurs in Python 2, a Py2Error will be thrown by the
client. The Python 2 exception is stored as the exception attribute of the
Py2Error object. The underlying traceback is attached to the Python 2
exception as the __traceback__ attribute.
>>> py2.int('asdf')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
...
python2.client.exceptions.Py2Error: ValueError: invalid literal for int() with base 10: 'asdf'
When you’re done using Python 2, you can end the session by calling the
Python2.shutdown() method. You can also use the Python2 object as a
context manager to automatically do the same thing when exiting the context.
>>> py2.shutdown()


Testing
This package uses Tox for testing. Tests are not included in the Python dist,
so you will need to clone the repo to run them. To run the unit tests, install
Tox and run the following command from the project’s base directory:
tox
After running tox, you can run the client-server integration tests with the
‘integration_tests.sh’ script. This script takes two arguments specifying the
Tox virtualenvs to use for Python 2 and 3, respectively:
./integration_tests.sh py27 py36
To modify the behavior of Tox, you can set the PYTEST_ADDOPTS variable.
For example, you can set the -x flag to abort after the first test
failure:
export PYTEST_ADDOPTS=-x
You can use the -n NUM flag to parallelize the tests using the
pytest-xdist plugin This adds some overhead to the test setup, so this
option is primarily useful for speeding up the integration tests.

export PYTEST_ADDOPTS=’-n 4’



Caveats

Supported types
Projection is only supported for basic builtin types. Other objects cannot be
projected to Python 2. The supported types are: bool, int, float,
complex, bytes, unicode, bytearray, range, slice,
list, tuple, set, frozenset, and dict. The None,
NotImplemented, and Ellipsis singletons are also supported.
In particular, Python 3 functions, types, and instances of user-defined classes
cannot currently be projected into Python 2.


Type introspection
The Py2Object class implements many “magic methods” from the Python 3 data
model. As a result, a Py2Object appears to be callable, iterable, etc.,
even if the underlying object is not. Attempting to perform such operations may
result in a Py2Error.
If you need to introspect a Python 2 object, use the corresponding Python 2
builtin functions. For example:
>>> i = py2.project(1)
>>> py2.callable(i)
<Py2Object False>
>>> py2.isinstance(i, py2.int)
<Py2Object True>


String types
In Python 2, str objects are raw byte strings, while in Python 3 they are
Unicode strings. This can lead to some confusion, as projecting a Python 3
str will result in a Python 2 unicode object, while lifting a Python 2
str will return a Python 3 bytes object.

>>> py2.project('foo')
<Py2Object u'foo'>
>>> py2.lift(py2.str(123))
b'123'




Division
The behavior of the division operator changed with PEP 238. This created
two alternate division operations, “true division” and “classic division”.
Classic division was removed in Python 3.
To respect this change, when two Py2Object s are divided, classic division
is used. When a Py2Object divides or is divided by a Python 3 value, true division is used.
>>> i = py2.project(1)
>>> j = py2.project(2)
>>> i / j # classic division
<Py2Object 0>
>>> i / 2 # true division
<Py2Object 0.5>
>>> 1 / j # true division
<Py2Object 0.5>



Further discussion

How it works
When you launch a Python 2 session, the library spawns a child process running
Python 2. This child process runs a server that listens for commands from
the Python 3 client. For each command, the server performs an operation in
Python 2 and returns the result either as an encoded value made up of
supported types, or a reference to a Python 2 object stored on the server.
On the client side, the library wraps Python 2 references with the
Py2Object class. This class implements many of the “magic methods” of the
Python 3 data model by sending commands to the Python 2 server to perform
the appropriate operation on the underlying Python 2 object.


Call-by-value semantics
When projecting a value or calling a Python 2 function with Python 3 arguments,
the arguments will be passed to Python 2 “by value”, that is, by encoding the
value of the argument to be decoded by the server. When using a Python 2
object, the object is stored in the Python 2 session and is passed “by
reference”.
This has some implications for the semantics of Python 2 functions. Suppose we
have a Python 2 function that mutates a list. If we pass this function a
Python 3 list, the list will be copied into Python 2 and the copy will be
mutated, but the original will not be modified:
>>> f = py2.eval("lambda l: l.append(1)")
>>> l = []
>>> f(l)
<Py2Object None>
>>> l
[]
However, if we project the list into Python 2 before passing it to the
function, then we can observe the modifications on the projected list:
>>> py2_l = py2.project(l)
>>> f(py2_l)
<Py2Object None>
>>> py2_l
<Py2Object [1]>


Return semantics
Returning generally occurs by reference except for operations that require a
specific return type (str(), int(), etc.). The main reason for this is
that returning by value may lose information about object identity that needs
to be preserved. Return values can be easily lifted to Python 2 if desired.


Object identity and lifespan
Each Python 2 object returned by the server is represented by a unique
Py2Object. This means that the is operator can be used to determine if
two Py2Object s refer to the same underlying object.
The Python 2 server stores all objects it returns, to prevent them from being
deallocated. When the corresponding Py2Object is deallocated in the Python
3 process, the underlying Python 2 object will be removed from the server cache
to allow it to be deallocated as appropriate.


Encoding algorithm
This library uses a simple JSON encoding for supported types. For a given
function call, each unique object will only be encoded once. This means that
data structures with circular references are supported. For a detailed
description of the algorithm, see the python2.shared.codec module.



Possible improvements

Python 2 types
Currently there is a single type for Python 2 objects in Python 3,
Py2Object. An alternate strategy would be to dynamically create Python 3
classes for each Python 2 type encountered, and create proxy objects as
instances of these classes.
The main benefit of this change would be better type introspection for Python 2
objects (see the discussion at Type introspection). However, it would be
more cumbersome and incur a performance cost, since the client would need to
know the type of each object and the methods supported by that type.
Additionally, this approach would not fully support the dynamic nature of the
Python type system, since the proxied type would not reflect changes to the
underlying type such as adding or removing methods.
This would require the server to return the object type for references, and
some mechanism for the client to introspect Python 2 types. The client would
cache types for the lifetime of the Python 2 session, with a mechanism to
explicitly refresh a type to pick up any changes that had occurred in Python 2.
Bootstrapping the type system might be a little tricky. We would want to
create a type Py2type such all proxy types are instances of, including
``Py2type`` itself. We would also probably want a base type for all proxy
objects, including types.


Python 3 proxy objects in Python 2
Currently the relationship between client and server is asymmetrical. The
client has a representation of Python 2 objects, but the server does not have
a way to represent Python 3 objects. We might like to add such a mechanism.
This would mean that instead of the simple request-response pattern from client
to server we have now, there would be the possibility of callbacks. In effect,
the two processes would act more like coroutines with the flow of control
passing back and forth between them.


Better Python version support
We could extend support to more Python 2 and 3 versions.



Similar projects
After writing this library, I discovered that I’m not the only one to have had
this idea. Sux is a library that provides similar functionality, with some
notable differences:

The library is much smaller and more lightweight, and only needs to be
installed in the Python 3 environment to work.
The main emphasis is on imports and function calls, which makes sense since
these are the most important operations for the using legacy packages. Most
other operators (e.g. arithmetic operators) are not supported.
The library uses Pickle to communicate between the Python 2 and 3 processes.
This is a good idea and I should probably have done the same, although I had
fun implementing the current encoding algorithm.