文章目錄
  1. 1. 装饰器
    1. 1.1. 使用functools
    2. 1.2. 带参数的装饰器
    3. 1.3. 类装饰器
  2. 2. metaclass

前些天在看David Beazley的Python3 metaprogramming视频,觉得是时候总结下视频中学到的内容, 这篇文章主要是这个视频的笔记,以及关于metaclass的一些思考.

装饰器

首先看一个需求,就是想在函数被调用时,记录一下,可以简单的加个print

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
def add(x, y):
print('add')
return x + y

def sub(x, y):
print('sub')
return x - y

def mul(x, y):
print('mul')
return x * y

def div(x, y):
print('div')
return x / y

print(add(3, 5))
print(sub(5, 2))

输出
add
8
sub
3

但是这样的话, print语句就重复了, 每个函数里都得加print, 于是我们可以使用装饰器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# 一个简单的装饰器
def debug(func):
def wrapper(*args, **kwargs):
print(func.__name__)
return func(*args, **kwargs)
return wrapper

@debug
def add(x, y):
return x + y

@debug
def sub(x, y):
return x - y

@debug
def mul(x, y):
return x * y

@debug
def div(x, y):
return x / y

print(add(3, 5))
print(sub(5, 2))

输出
add
8
sub
3

使用functools

但是这个简单的装饰器是存在问题的,它会忽略被装饰的函数

1
2
3
4
print(add)

输出
<function debug.<locals>.wrapper at 0x1037a6bf8>

这里add函数经过debug装饰器装饰后,函数名都被忽略了,这个时候functools模块就派上用场了, 里面的wraps装饰器就是用来解决这个问题

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
from functools import wraps

def debug(func):
msg = func.__qualname__
@wraps(func)
def wrapper(*args, **kwargs):
print(msg)
return func(*args, **kwargs)
return wrapper

@debug
def add(x, y):
return x + y

@debug
def sub(x, y):
return x - y

@debug
def mul(x, y):
return x * y

@debug
def div(x, y):
return x / y

print(add(3, 5))
print(sub(5, 2))

输出
add
8
sub
3

print(add)
输出
<function add at 0x1037afa60>

带参数的装饰器

有时候装饰器里想传入一些参数, 这时就可以写带参数的装饰器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# decorators with args
from functools import wraps

def debug(prefix=''):
def decroate(func):
msg = prefix + func.__qualname__
@wraps(func)
def wrapper(*args, **kwargs):
print(msg)
return func(*args, **kwargs)
return wrapper
return decroate

@debug('###')
def add(x, y):
return x + y

print(add(3, 2))
输出
###add
5

这种带参数的装饰器有一个头疼的问题是, 使用装饰器时, 如果不想传参数, 也得加上括号, 不然会报错

1
2
3
4
5
6
7
8
9
10
11
12
13
14
@debug
def sub(x, y):
return x - y

sub(5, 3)

---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-40-9a93be17056c> in <module>
3 return x - y
4
----> 5 sub(5, 3)

TypeError: decroate() takes 1 positional argument but 2 were given

加上括号后, 就不报错, 这很丑陋

1
2
3
4
5
@debug()
def sub(x, y):
return x - y

print(sub(5, 3))

这里有一个小技巧, 实现如下

1
2
3
4
5
6
7
8
9
10
11
from functools import wraps, partial

def debug(func=None, prefix=''):
if func is None:
return partial(debug, prefix=prefix)
msg = prefix + func.__qualname__
@wraps(func)
def wrapper(*args, **kwargs):
print(msg)
return func(*args, **kwargs)
return wrapper

如此, 当使用默认参数时, 即便不带括号时,也不会报错

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
@debug(prefix='###')
def add(x, y):
return x + y


@debug
def sub(x, y):
return x - y

print(add(3, 2))
print(sub(5, 3))
输出
###add
5
sub
2

类装饰器

以下我们定义一个Spam类,

1
2
3
4
5
6
7
8
9
10
11
class Spam:
def a(self):
pass
def b(self):
pass
@classmethod
def c(cls):
pass
@staticmethod
def d():
pass

然后我们想在类的方法被调用时, 能够记录下, 想上面的函数被调用时一样, 这时我们就会可以编写一个类装饰器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
def debugmethods(cls):
# cls is class
for key, val in vars(cls).items():
if callable(val):
setattr(cls, key, debug(val))
return cls

@debugmethods
class Spam:
def a(self):
pass
def b(self):
pass
@classmethod
def c(cls):
pass
@staticmethod
def d():
pass

spam = Spam()
spam.a()
spam.b()
spam.c()
spam.d()
输出
Spam.a
Spam.b

这里只打印了a和b, 没有打印c和d, 这什么原因呢? 这是因为classmethod和staticmethod都是descriptor, 也就是描述器, 它们没有实现__call__方法,也就不是callable的

我们也可以编写一个类装饰器, 当获取一个属性时, 打印日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# debug access
def debugattr(cls):
orig_getattribute = cls.__getattribute__
def __getattribute__(self, name):
print('Get:', name)
return orig_getattribute(self, name)
cls.__getattribute__ = __getattribute__
return cls

@debugattr
class Spam:
def __init__(self, x, y):
self.x = x
self.y = y

spam = Spam(2, 3)
print(spam.x)
输出
Get: x
2

metaclass

现在我们想对所有的类都能打印日志,一个解决的办法是在所有的类前面都加上类装饰器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
@debugmethods
class Base:
def a(self):
pass
def b(self):
pass

@debugmethods
class Spam(Base):
def a(self):
pass
b = Base()
b.a()
s = Spam()
s.a()
输出
Base.a
Spam.a

但这样很麻烦,于是metaclass派上用场了, metaclass最强的地方是可以控制类的创建

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# a metaclass
class debugmeta(type):
def __new__(cls, clsname, bases, clsdict):
clsobj = super().__new__(cls, clsname, bases, clsdict)
# print("here", cls, clsname, type(clsobj), clsobj)
clsobj = debugmethods(clsobj)
# print(vars(clsobj))
return clsobj

class Base(metaclass=debugmeta):
def a(self):
pass
def b(self):
pass


class Spam(Base):
def __init__(self, name):
self.name = name

def a(self):
pass

b = Base()
b.a()
s = Spam('name')
s.a()

输出
Base.a
Spam.__init__
Spam.a

从上面的例子中我们看到,有一个类有metaclass, 它的所有子类都有metaclass, 这说明metaclass是会被继承的。结合蔡元楠的《metaclass, 是潘多拉魔盒还是阿拉丁神灯》, 可以知道,这里debugmeta其实不只一种写法,在__init__函数里实现也是可以的。重载__init__的实现如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# a metaclass
class debugmeta(type):
def __init__(cls, name, bases, kwds):
super(debugmeta, cls).__init__(name, bases, kwds)
cls = debugmethods(cls)

class Base(metaclass=debugmeta):
def a(self):
pass
def b(self):
pass


class Spam(Base):
def __init__(self, name):
self.name = name

def a(self):
pass
b = Base()
b.a()
s = Spam('lala')
s.a()
输出
Base.a
Spam.__init__
Spam.a

这是因为, 所有的类都是type的实例, 都是对type的__call__方法进行重载, 而type的__call__方法会调用type.__new__(typeclass, classname, superclasses, attributedict)type.__init__(class, classname, superclasses, attributedict), 所以上面重写__new__和重写__init__都是可以的。

学到这里,我脑海里冒出了一个想法,就是为啥这里一定要用metaclass呢? 用继承的方式难道不行吗?于是自己尝试写了个继承的方式, 发现也是跑得通的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# a baseclass
class debugmeta:
def __new__(cls, *args, **kwargs):
cls = debugmethods(cls)
clsobj = object.__new__(cls)
# clsobj = debugmethods(clsobj)
# print('aaa', type(clsobj), clsobj)
# print('aaa', clsobj, type(clsobj), vars(clsobj), dir(clsobj), type(clsobj.a), callable(clsobj.a))
return clsobj

class Base(debugmeta):
def a(self):
pass


class Spam(Base):
def __init__(self, name):
self.name = name
def a(self):
pass


b = Base()
b.a()
s = Spam('name')
s.a()
输出
Base.a
Spam.__init__
Spam.a

但实际上,这样做法是有问题的,后面等到后面我们来纠正这个问题。

既然如此,那么蔡元楠在《metaclass, 是潘多拉魔盒还是阿拉丁神灯》介绍的yaml的动态序列化和逆序列化的能力又为何要用metaclass实现呢?用继承难道不行吗?于是也写了一个继承的版本, 代码如下。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
import yaml


class MyYAMLObjectBaseclass(object):
"""
The metaclass for YAMLObject.
"""

def __new__(cls, *args, **kwargs):
if cls.yaml_tag:
cls.yaml_loader.add_constructor(cls.yaml_tag, cls.from_yaml)
cls.yaml_dumper.add_representer(cls, cls.to_yaml)
return object.__new__(cls)

class MyYAMLObject(MyYAMLObjectBaseclass):
"""
An object that can dump itself to a YAML stream
and load itself from a YAML stream.
"""


__slots__ = () # no direct instantiation, so allow immutable subclasses

yaml_loader = yaml.Loader
yaml_dumper = yaml.Dumper

yaml_tag = None
yaml_flow_style = None

@classmethod
def from_yaml(cls, loader, node):
"""
Convert a representation node to a Python object.
"""

return loader.construct_yaml_object(node, cls)

@classmethod
def to_yaml(cls, dumper, data):
"""
Convert a Python object to a representation node.
"""

return dumper.represent_yaml_object(cls.yaml_tag, data, cls,
flow_style=cls.yaml_flow_style)


class Monster(MyYAMLObject):
yaml_tag = '!Monster'

def __init__(self, name, hp, ac, attacks):
self.name = name
self.hp = hp
self.ac = ac
self.attacks = attacks

def __repr__(self):
return "{}(name={}, hp={}, ac={}, attacks={}".format(self.__class__.__name__, self.name, self.hp,
self.ac, self.attacks)


class Dragon(Monster):
yaml_tag = '!Dragon'

def __init__(self, name, hp, ac, attacks, energy):
super(Dragon, self).__init__(name, hp, ac, attacks)
self.energy = energy

def __repr__(self):
return "{}(name={}, hp={}, ac={}, attacks={}, energy={})".format(self.__class__.__name__, self.name, self.hp,
self.ac, self.attacks, self.energy)

m = Monster(name='Cave spider', hp=[2, 6], ac=16, attacks=['BITE', 'HURT'])
ms = yaml.dump(m)
# print(ms)
d = Dragon(name='Cave spider', hp=[2, 6], ac=17, attacks=['BITE', 'HURT'], energy=5000)
ds = yaml.dump(d)
# print(ds)

print(yaml.load(ms, Loader=yaml.Loader))
print(yaml.load(ds, Loader=yaml.Loader))
输出
Monster(name=Cave spider, hp=[2, 6], ac=16, attacks=['BITE', 'HURT']
Dragon(name=Cave spider, hp=[2, 6], ac=17, attacks=['BITE', 'HURT'], energy=5000)

所以其实到这里我还是没有明白为啥yaml要用metaclass来实现这个功能, 直到把子类的yaml_tag去掉, 才发现问题。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
import yaml


class Monster(MyYAMLObject):
yaml_tag = '!Monster'

def __init__(self, name, hp, ac, attacks):
self.name = name
self.hp = hp
self.ac = ac
self.attacks = attacks

def __repr__(self):
return "{}(name={}, hp={}, ac={}, attacks={}".format(self.__class__.__name__, self.name, self.hp,
self.ac, self.attacks)


class Dragon(Monster):

def __init__(self, name, hp, ac, attacks, energy):
super(Dragon, self).__init__(name, hp, ac, attacks)
self.energy = energy

def __repr__(self):
return "{}(name={}, hp={}, ac={}, attacks={}, energy={})".format(self.__class__.__name__, self.name, self.hp,
self.ac, self.attacks, self.energy)

m = Monster(name='Cave spider', hp=[2, 6], ac=16, attacks=['BITE', 'HURT'])
ms = yaml.dump(m)
# print(ms)
d = Dragon(name='Cave spider', hp=[2, 6], ac=17, attacks=['BITE', 'HURT'], energy=5000)
ds = yaml.dump(d)
# print(ds)

print(yaml.load(ms, Loader=yaml.Loader))
print(yaml.load(ds, Loader=yaml.Loader))
输出
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-58-1e3c4c44fbaf> in <module>
33 # print(ds)
34
---> 35 print(yaml.load(ms, Loader=yaml.Loader))
36 print(yaml.load(ds, Loader=yaml.Loader))

<ipython-input-58-1e3c4c44fbaf> in __repr__(self)
24 def __repr__(self):
25 return "{}(name={}, hp={}, ac={}, attacks={}, energy={})".format(self.__class__.__name__, self.name, self.hp,
---> 26 self.ac, self.attacks, self.energy)
27
28 m = Monster(name='Cave spider', hp=[2, 6], ac=16, attacks=['BITE', 'HURT'])

AttributeError: 'Dragon' object has no attribute 'energy'

这是因为Dragon这里没有yaml_tag的时候, 会继承父类Monster的yaml_tag, 这时Dragon.yaml_tag就是非空, 然后就会将!Monster这个标记与Dragon绑定到一起了, 覆盖了前面!Monster与Monster的绑定, 这时再去加载Monster类dump出来的内容, 就会报没有energy. 而使用metaclass就不存在这个问题, 因为在创建Dragon类时, 传入的属性字典里不会带有yaml_tag, 也就不会将!Monster这个标记与Dragon绑定到一起. 写代码测试如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
import yaml


class MyYAMLObjectMetaclass(type):
"""
The metaclass for YAMLObject.
"""

def __new__(cls, name, bases, kwds):
print(cls, name, bases, kwds.get('yaml_tag', 'hahaha'))
clsobj = super().__new__(cls, name, bases, kwds)
if 'yaml_tag' in kwds and kwds['yaml_tag'] is not None:
clsobj.yaml_loader.add_constructor(clsobj.yaml_tag, clsobj.from_yaml)
clsobj.yaml_dumper.add_representer(clsobj, clsobj.to_yaml)
return clsobj

class ThisYAMLObjectMetaclass(type):
"""
The metaclass for YAMLObject.
"""

def __init__(cls, name, bases, kwds):
print(cls, name, bases, kwds.get('yaml_tag', 'hahaha'))
super(ThisYAMLObjectMetaclass, cls).__init__(name, bases, kwds)
if 'yaml_tag' in kwds and kwds['yaml_tag'] is not None:
cls.yaml_loader.add_constructor(cls.yaml_tag, cls.from_yaml)
cls.yaml_dumper.add_representer(cls, cls.to_yaml)


class ThisYAMLObjectBaseclass(object):
"""
The metaclass for YAMLObject.
"""

def __new__(cls, *args, **kwargs):
print('base __new__', cls, args, kwargs)
if cls.yaml_tag:
cls.yaml_loader.add_constructor(cls.yaml_tag, cls.from_yaml)
cls.yaml_dumper.add_representer(cls, cls.to_yaml)
return object.__new__(cls)


class MyYAMLObject(metaclass=ThisYAMLObjectMetaclass):
# class MyYAMLObject(ThisYAMLObjectBaseclass):
# class MyYAMLObject(metaclass=MyYAMLObjectMetaclass):
"""
An object that can dump itself to a YAML stream
and load itself from a YAML stream.
"""


__slots__ = () # no direct instantiation, so allow immutable subclasses

yaml_loader = yaml.Loader
yaml_dumper = yaml.Dumper

yaml_tag = None
yaml_flow_style = None

@classmethod
def from_yaml(cls, loader, node):
"""
Convert a representation node to a Python object.
"""

return loader.construct_yaml_object(node, cls)

@classmethod
def to_yaml(cls, dumper, data):
"""
Convert a Python object to a representation node.
"""

return dumper.represent_yaml_object(cls.yaml_tag, data, cls,
flow_style=cls.yaml_flow_style)


class Monster(MyYAMLObject):
yaml_tag = '!Monster'

def __init__(self, name, hp, ac, attacks):
self.name = name
self.hp = hp
self.ac = ac
self.attacks = attacks

def __repr__(self):
return "{}(name={}, hp={}, ac={}, attacks={}".format(self.__class__.__name__, self.name, self.hp,
self.ac, self.attacks)


class Dragon(Monster):
# yaml_tag = '!Dragon'

def __init__(self, name, hp, ac, attacks, energy):
super(Dragon, self).__init__(name, hp, ac, attacks)
self.energy = energy

def __repr__(self):
return "{}(name={}, hp={}, ac={}, attacks={}, energy={})".format(self.__class__.__name__, self.name, self.hp,
self.ac, self.attacks, self.energy)

m = Monster(name='Cave spider', hp=[2, 6], ac=16, attacks=['BITE', 'HURT'])
ms = yaml.dump(m)
# print(ms)
d = Dragon(name='Cave spider', hp=[2, 6], ac=17, attacks=['BITE', 'HURT'], energy=5000)
ds = yaml.dump(d)
# print(ds)

print(yaml.load(ms, Loader=yaml.Loader))
print(yaml.load(ds, Loader=yaml.Loader))
输出
<class '__main__.MyYAMLObject'> MyYAMLObject () {'__module__': '__main__', '__qualname__': 'MyYAMLObject', '__doc__': '\n An object that can dump itself to a YAML stream\n and load itself from a YAML stream.\n ', '__slots__': (), 'yaml_loader': <class 'yaml.loader.Loader'>, 'yaml_dumper': <class 'yaml.dumper.Dumper'>, 'yaml_tag': None, 'yaml_flow_style': None, 'from_yaml': <classmethod object at 0x1038e9320>, 'to_yaml': <classmethod object at 0x1038e9390>}
<class '__main__.Monster'> Monster (<class '__main__.MyYAMLObject'>,) {'__module__': '__main__', '__qualname__': 'Monster', 'yaml_tag': '!Monster', '__init__': <function Monster.__init__ at 0x1038d2f28>, '__repr__': <function Monster.__repr__ at 0x1038ce048>}

<class '__main__.Dragon'> Dragon (<class '__main__.Monster'>,) {'__module__': '__main__', '__qualname__': 'Dragon', '__init__': <function Dragon.__init__ at 0x1038ce0d0>, '__repr__': <function Dragon.__repr__ at 0x1038ce158>, '__classcell__': <cell at 0x103837af8: ThisYAMLObjectMetaclass object at 0x7fc8fb130de8>}
Monster(name=Cave spider, hp=[2, 6], ac=16, attacks=['BITE', 'HURT']

Dragon(name=Cave spider, hp=[2, 6], ac=17, attacks=['BITE', 'HURT'], energy=5000)

从上面的结果里可以看到, yaml_tag是没有在Dragon类的属性字典里的,即便是Dragon类会从Monster那里继承yaml_tag.

回到前面的用基类来实现对所有类使用debugmethods进行装饰的例子。这里因为每次创建对象的时候都会调用__new__方法, 会导致多次调用debugmethods装饰器, 这样会导致创建多少个对象, 调用一次类的方法就会输有多次, 测试如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# a baseclass
class debugmeta:
def __new__(cls, *args, **kwargs):
cls = debugmethods(cls)
clsobj = object.__new__(cls)
# clsobj = debugmethods(clsobj)
# print('aaa', type(clsobj), clsobj)
# print('aaa', clsobj, type(clsobj), vars(clsobj), dir(clsobj), type(clsobj.a), callable(clsobj.a))
return clsobj

class Base(debugmeta):
def a(self):
pass


class Spam(Base):
def __init__(self, name):
self.name = name
def a(self):
pass


b = Base()
b.a()
s = Spam('name')
s.a()
print('-------')
s = Spam('lblb')
s.a()
输出如下
here <class '__main__.Base'>
Base.a
here <class '__main__.Spam'>
Spam.__init__
Spam.a
-------
here <class '__main__.Spam'>
Spam.__init__
Spam.__init__
Spam.a
Spam.a

这里Spam类创建了两个对象, 就调用了两次debugmethods, 所以会有多次输出。

到了这里, 我终于明白为什么要用metaclass来解决对于所有类使用debugmethods来装饰的问题,因为在metaclass里实现,则只会调用一次,因为一个类的创建只需要一次。也明白为什么yaml要使用metaclass, 而不是继承了。

总的来说, metaclass并不是什么奇淫巧技,简单来说就是一种改变类创建过程的能力。当然, 绝大多数情况下都不需要用到它。

打赏作者

文章目錄
  1. 1. 装饰器
    1. 1.1. 使用functools
    2. 1.2. 带参数的装饰器
    3. 1.3. 类装饰器
  2. 2. metaclass