Current time: %s' % datetime.datetime.utcnow().strftime('%H:%M') u'
Current time: 23:52'
```
The string passed to `strftime` is native (so bytes in 2.x and Unicode
in 3.x). The return value is a native string again and ASCII only.
As such both on 2.x and 3.x it will be a Unicode string once string
formatted.
- `u'foo'` always refers to a Unicode string. Many libraries already
had pretty excellent Unicode support in 2.x so that literal should not
be surprising to many.
- `b'foo'` always refers to something that can hold arbitrary bytes.
Since 2.6 does not actually have a `bytes` object like Python 3.3
has and Python 3.3 lacks an actual bytestring the usefulness of this
literal is indeed a bit limited. It becomes immediately more useful
when paired with the `bytearray` object which has the same interface
on 2.x and 3.x:
```pycon
>>> bytearray(b' foo ').strip()
bytearray(b'foo')
```
Since it's also mutable it's quite efficient at modifying raw bytes
and you can trivially convert it to something more conventional by
wrapping the final result in `bytes()` again.
In addition to these basic rules I also add `text_type`, `unichr`
and `string_types` variables to my compatibility module as shown above.
With those available the big changes are:
- `isinstance(x, basestring)` becomes `isinstance(x, string_types)`.
- `isinstance(x, unicode)` becomes `isinstance(x, text_type)`.
- `isinstance(x, str)` with the intention of catching bytes becomes
`isinstance(x, bytes)` or `isinstance(x, (bytes, bytearray))`.
I also created a `implements_to_string` class decorator that helps
implementing classes with `__unicode__` or `__str__` methods:
```python
if PY2:
def implements_to_string(cls):
cls.__unicode__ = cls.__str__
cls.__str__ = lambda x: x.__unicode__().encode('utf-8')
return cls
else:
implements_to_string = lambda x: x
```
The idea is that you just implement `__str__` on both 2.x and 3.x and
let it return Unicode strings (yes, looks a bit odd in 2.x) and the
decorator automatically renames it to `__unicode__` for 2.x and adds a
`__str__` that invokes `__unicode__` and encodes the return value to
utf-8. This pattern has been pretty common in the past with 2.x modules.
For instance Jinja2 and Django use it.
Here is an example for the usage:
```python
@implements_to_string
class User(object):
def __init__(self, username):
self.username = username
def __str__(self):
return self.username
```
## Metaclass Syntax Changes
Since Python 3 changed the syntax for defining the metaclass to use in an
incompatible way this makes porting a bit harder than it should be. Six
has a `with_metaclass` function that can work around this issue but it
generates a dummy class that shows up in the inheritance tree. For Jinja2
I was not happy enough with that solution and modified it a bit. The
external API is the same but the implementation uses a temporary class
to hook in the metaclass. The benefit is that you don't have to pay a
performance penalty for using it and your inheritance tree stays nice.
The code is a bit hard to understand. The basic idea is exploiting the
idea that metaclasses can customize class creation and are picked by by
the parent class. This particular implementation uses a metaclass to
remove its own parent from the inheritance tree on subclassing. The end
result is that the function creates a dummy class with a dummy metaclass.
Once subclassed the dummy classes metaclass is used which has a
constructor that basically instances a new class from the original parent
and the actually intended metaclass. That way the dummy class and dummy
metaclass never show up.
This is what it looks like:
```python
def with_metaclass(meta, *bases):
class metaclass(meta):
__call__ = type.__call__
__init__ = type.__init__
def __new__(cls, name, this_bases, d):
if this_bases is None:
return type.__new__(cls, name, (), d)
return meta(name, bases, d)
return metaclass('temporary_class', None, {})
```
And here is how you use it:
```python
class BaseForm(object):
pass
class FormType(type):
pass
class Form(with_metaclass(FormType, BaseForm)):
pass
```
## Dictionaries
One of the more annoying changes in Python 3 are the changes on the
dictionary iterator protocols. In Python 2 all dictionaries had
`keys()`, `values()` and `items()` that returned lists and
`iterkeys()`, `itervalues()` and `iteritems()` that returned
iterators. In Python 3 none of that exists any more. Instead they were
replaced with new methods that return view objects.
`keys()` returns a key view which behaves like some sort of read-only
set, `values()` which returns a read-only container and iterable (not an
iterator!) and `items()` which returns some sort of read-only set-like
object. Unlike regular sets it however can also point to mutable objects
in which case some methods will fail at runtime.
On the positive side a lot of people missed that views are not iterators
so in many cases you can just ignore that. Werkzeug and Django implement
a bunch of custom dictionary objects and in both cases the decision was
made to just ignore the existence of view objects and let `keys()` and
friends return iterators.
This is currently the only sensible thing to do due to limitations of the
Python interpreter. There are a few problems with it:
- The fact that the views are not iterators by themselves mean that in
the average case you create a temporary object for no good reason.
- The set-like behavior of the builtin dictionary views cannot be
replicated in pure Python due to [limitations in the interpreter](http://bugs.python.org/issue2226).
- Implementing views for 3.x and iterators for 2.x would mean a lot of
code duplication.
This is what the Jinja2 codebase went with for iterating over
dictionaries:
```python
if PY2:
iterkeys = lambda d: d.iterkeys()
itervalues = lambda d: d.itervalues()
iteritems = lambda d: d.iteritems()
else:
iterkeys = lambda d: iter(d.keys())
itervalues = lambda d: iter(d.values())
iteritems = lambda d: iter(d.items())
```
For implementing dictionary like objects a class decorator can become
useful again:
```python
if PY2:
def implements_dict_iteration(cls):
cls.iterkeys = cls.keys
cls.itervalues = cls.values
cls.iteritems = cls.items
cls.keys = lambda x: list(x.iterkeys())
cls.values = lambda x: list(x.itervalues())
cls.items = lambda x: list(x.iteritems())
return cls
else:
implements_dict_iteration = lambda x: x
```
In that case all you need to do is to implement the `keys()` and friends
method as iterators and the rest happens automatically:
```python
@implements_dict_iteration
class MyDict(object):
...
def keys(self):
for key, value in iteritems(self):
yield key
def values(self):
for key, value in iteritems(self):
yield value
def items(self):
...
```
## General Iterator Changes
Since iterators changed in general a bit of help is needed to make this
painless. The only change really is the transition from `next()` to
`__next__`. Thankfully this is already transparently handled. The only
thing you really need to change is to go from `x.next()` to `next(x)`
and the language does the rest.
If you plan on defining iterators a class decorator again becomes helpful:
```python
if PY2:
def implements_iterator(cls):
cls.next = cls.__next__
del cls.__next__
return cls
else:
implements_iterator = lambda x: x
```
For implementing this class just name the iteration step method
`__next__` in all versions:
```python
@implements_iterator
class UppercasingIterator(object):
def __init__(self, iterable):
self._iter = iter(iterable)
def __iter__(self):
return self
def __next__(self):
return next(self._iter).upper()
```
## Transformation Codecs
One of the nice features of the Python 2 encoding protocol was that it was
independent of types. You could register an encoding that would transform
a csv file into a numpy array if you would have preferred that. This
feature however was not well known since the primary exposed interface of
encodings was attached to string objects. Since they got stricter in 3.x
a lot of that functionality was removed in 3.0 but later reintroduced in
3.3 again since it proved useful. Basically all codecs that did not
convert between Unicode and bytes or the other way round were unavailable
until 3.3. Among those codecs are the hex and base64 codec.
There are two use cases for those codecs: operations on strings and
operations on streams. The former is well known as `str.encode()` in
2.x but now looks different if you want to support 2.x and 3.x due to the
changed string API:
```pycon
>>> import codecs
>>> codecs.encode(b'Hey!', 'base64_codec')
'SGV5IQ==\n'
```
You will also notice that the codecs are missing the aliases in 3.3 which
requires you to write `'base64_codec'` instead of `'base64'`.
(These codecs are preferable over the functions in the `binascii` module
because they support operations on streams through the [incremental
encoding and decoding support](http://docs.python.org/3/library/codecs.html#incrementalencoder-objects).)
## Other Notes
There are still a few places where I don't have nice solutions for yet or
are generally annoying to deal with but they are getting fewer. Some of
them are unfortunately now part of the Python 3 API are hard to discover
until you trigger an edge case.
- Filesystem and file IO access continues to be annoying to deal with on
Linux due to it not being based on Unicode. The `open()` function
and the filesystem layer have dangerous platform specific defaults.
If I SSH into a `en_US` machine from an `de_AT` one for instance
Python loves falling back to ASCII encoding for both file system and
file operations.
Generally I noticed the most reliable way to do text on Python 3 that
also works okay on 2.x is just to open files in binary mode and
explicitly decode. Alternatively you can use the `codecs.open` or
`io.open` function on 2.x and the builtin `open` on Python 3 with
an explicit encoding.
- URLs in the standard library are represented incorrectly as Unicode
which causes some URLs to not be dealt with correctly on 3.x.
- Raising exceptions with a traceback object requires a helper function
since the syntax changed. This is very uncommon in general and easy
enough to wrap. Since the syntax changed this is one of the
situations where you will have to move code into an exec block:
```python
if PY2:
exec('def reraise(tp, value, tb):\n raise tp, value, tb')
else:
def reraise(tp, value, tb):
raise value.with_traceback(tb)
```
- The previous `exec` trick is useful in general if you have some code
that depends on different syntax. Since exec itself has a different
syntax now you won't be able to use it to execute something against an
arbitrary namespace. This is not a huge deal because `eval` with
`compile` can be used as a drop-in that works on both versions.
Alternatively you can bootstrap an `exec_` function through `exec`
itself.
```python
exec_ = lambda s, *a: eval(compile(s, '