.. .rst: expy tutorial file, created by Yingjie Lan, Thu July 9, 2009 Create Extension Types ======================= #. :ref:`diveinex` #. :ref:`classdef` #. :ref:`members` #. :ref:`class_members` #. :ref:`methods` #. :ref:`specials` #. :ref:`property` #. :ref:`gentest` #. :ref:`basetype` #. :ref:`numeric` #. :ref:`mapping` #. :ref:`sequence` #. :ref:`iterator` #. :ref:`memindex` #. :ref:`exposure` #. :ref:`tp_init` #. :ref:`INCREF` #. :ref:`conclusion` .. _diveinex: A Dive-in Example ----------------- In the example below, the module only has a class extension: .. literalinclude:: module3.py :linenos: The explanation of this example follows. .. _classdef: Defining A New Extension Type ------------------------------- Line 6 start the definition of your extension type: the main thing you should notice is that it extends *public*, which is a class defined in the expy module. This basically says that this class should be totally visible in Python. .. _members: Adding Instance Members to Types -------------------------------- Line 8-9 declares two instance members, by using the *ifield* (i.e.: instance field) class provided in the expy module. You provide the type, the access (such as public or private), the document string, and you can also provide the flag (must be a string), which corresponds to the flags field of the PyMemberDef structure in Python C API. In Line 8, the *_age* member is of 'int' type, and it is a private member. In Line 9, the *name* member is of 'str' type (which in C it corresponds to 'char\*'), and is 'readonly', as indicated by the "RO" flag. The following flag constants are defined in :file:`structmember.h`; they may be combined using bitwise-OR. +---------------------------+----------------------------------------------+ | Constant | Meaning | +===========================+==============================================+ | :const:`READONLY` | Never writable. | +---------------------------+----------------------------------------------+ | :const:`RO` | Shorthand for :const:`READONLY`. | +---------------------------+----------------------------------------------+ | :const:`READ_RESTRICTED` | Not readable in restricted mode. | +---------------------------+----------------------------------------------+ | :const:`WRITE_RESTRICTED` | Not writable in restricted mode. | +---------------------------+----------------------------------------------+ | :const:`RESTRICTED` | Not readable or writable in restricted mode. | +---------------------------+----------------------------------------------+ .. index:: single: READONLY single: RO single: READ_RESTRICTED single: WRITE_RESTRICTED single: RESTRICTED An interesting advantage of using the *ifield()* to build fields/members at runtime is that any attribute defined this way can have an associated doc string simply by providing the text in the constructor. An application can use the introspection API to retrieve the descriptor from the class object, and get the doc string using its :attr:`__doc__` attribute. One more good thing about instance members:: The order in which these fields appear will be preserved in the generated object structure. .. _class_members: Adding Class Members to Types -------------------------------- Class members/attributes of extension types are read only, thus they are ideal for providing constants/data to users. Line 10 defines a class member/field, which is the version of the class. Instead of using an 'ifield' constructor, you use a 'cfield' (class field) constructor, which takes only one parameter, namely, the value you want to store. This parameter is specified in the same way as default values to your function arguments. This class field is accessed by *mate.version*, once you have built and imported the module. .. _methods: Adding Methods to Types -------------------------- Lines 19-29 defines a method named "rename" for the extension type. In Line 19, we use the decorator *imethod* to declare that this is an "instance method", whose return type is 'void'. The first argument will be the instance, the next argument is the new name, of type 'str' (which translated in C is 'char*'). It is very similar to how we defined functions for modules. The name mangling is done by appending the class name to the function name, connected by an underscore '_'. .. _specials: Adding Special Methods ------------------------ Lines 12-17 defines a special method "__init__", which initializes an instance. Those special methods does not need any decoration, they are automatically detected, because their return type, access type, etc, are known in advance. Similar to other methods, you can use @throws decorators. Unlike other special methods, you can specify arbitrary set of arguments to the __init__ method, just like you do with pure python classes. Other special methods (in the object protocal) that are recognized -- more can be added if anything is missing: .. code-block:: python :linenos: special_methods={ #returns: int, with 0:success, -1:failure '__init__':{'cat':'imethod', 'ret':int, 'acc':public}, #signature: def __new__(cls, args, kwds): '__new__':{'cat':'cmethod', 'ret':object, 'acc':public, 'nowrap':True}, #signature: def __del__(me): '__del__':{'cat':'imethod', 'ret':void, 'acc':public, 'nowrap':True}, #signature: def __call__(cls, args, kwds): '__call__':{'cat':'imethod', 'ret':object, 'acc':public, 'nowrap':True}, #signature: def __cmp__(me, u[=lambda:mytype]): [optional] '__cmp__':{'cat':'imethod', 'ret':int, 'acc':public, 'nowrap':True}, #signature: def __getattr__(me, name=str): '__getattr__':{'cat':'imethod', 'ret':object, 'acc':public, 'nowrap':True}, #signature: def __setattr__(me, name=str, attr=object): '__setattr__':{'cat':'imethod', 'ret':int, 'acc':public, 'nowrap':True}, #signature: def __getattro__(me, name): '__getattro__':{'cat':'imethod', 'ret':object, 'acc':public, 'nowrap':True}, #signature: def __setattro__(me, name, attr): '__setattro__':{'cat':'imethod', 'ret':int, 'acc':public, 'nowrap':True}, #signature: def __hash__(me): '__hash__':{'cat':'imethod', 'ret':long, 'acc':public, 'nowrap':True}, #signature: def __print__(me, f=raw_type('FILE*'), flags=int): '__print__':{'cat':'imethod', 'ret':int, 'acc':public, 'nowrap':True}, #signature: def __repr__(me): '__repr__':{'cat':'imethod', 'ret':object, 'acc':public, 'nowrap':True}, #signature: def __str__(me): '__str__':{'cat':'imethod', 'ret':object, 'acc':public, 'nowrap':True}, #signature: def __iter__(me): '__iter__':{'cat':'imethod', 'ret':object, 'acc':public, 'nowrap':True}, #signature: def __next__(me): '__next__':{'cat':'imethod', 'ret':object, 'acc':public, 'nowrap':True}, } You may refer to the Python C API to learn more about those special methods, and how to implement them. .. _property: Adding Properties ------------------- Lines 31-51 defines a property named 'age'. Tow functions 'get_age' and 'set_age' are defined, without decoration. They must conform to that signature so that the generated C function is a well defined getter and setter function for a property. Similar to other methods, you can use @throws decorators. The main feature provided here is to throw an exception if the age argument is of the wrong type or value. .. _gentest: Generate and Test the Example ------------------------------- Now we use the following setup code to generate and compile this module. .. literalinclude:: module3_setup.py :linenos: And here is the generated code with the header file: .. literalinclude:: module3.h :language: c :linenos: .. literalinclude:: module3.c :language: c :linenos: Note that for each extention type you create, you will get three macros for free: _Check, _CheckExact, and _NEW. Start a Python session and test what we have got here:: ylan@montop:~/expy/Doc$ python module3_setup.py build ylan@montop:~/expy/Doc$ cd build/lib.linux-i686-2.5/ ylan@montop:~/expy/Doc/build/lib.linux-i686-2.5$ python Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from module3 import mate >>> x = mate('John', 23) >>> x.name 'John' >>> x.age 23 >>> x.age = -23 Traceback (most recent call last): File "", line 1, in ValueError: Must be Positive >>> x.age = x Traceback (most recent call last): File "", line 1, in TypeError: Must be an Integer >>> x.age 23 >>> x.rename('Job') >>> x.name 'Job' .. _basetype: Create Base Extension Types --------------------------- If you have implemented your methods carefully, you can declare your extension types as a base type, so that somebody can extend it in pure Python. To declare that your type can serve as a base type, you simply do this:: class mybase(public, basetype): """this extension type can be a base type.""" def ameth(me, you): ... For more information on how to implement a base type, please refer to Python C API. .. _numeric: Create Numeric Types ------------------------------ Python allows one to create types/classes that can act like numbers, by implementing the numeric protocol. You can create a numeric extension type in similar ways as you create a pure python numeric type with expy-cxpy. There are, however, two distinct categories of numeric extension types in Python C API. Category 1: the special numeric type (by indicating that the second argument is also of the same type, see example below), whose numeric operations are only defined between an object of the defining type and an object of the same type; Category 2: the generic numeric type, whose numeric operations are defined between an object of the defining type and an arbitrary object (so the second argument is of the default 'object' type)). When you implement your numeric type, don't mix up these two categories, otherwise EXPY will complain. The following example is for Category 1. A new extension type 'counter' is created, it has an integer number, and a name. We only defined the __add__ operator, when two counters added together, not only their numbers are added, but also their names are merged. .. literalinclude:: module4.py :linenos: To indicate that the second argument to __add__ function must be of the same 'counter' type, we use 'lambda:counter' to specify its type. It might be more appealing to simply use something like '__add__(me, u=counter)', but since 'counter' is not defined yet, Python compiler would complain about this and abort, so the 'lambda' function is used here to *delay* the evaluation till code generation. To enforce the type restriction, we can add a @throws decorator to check on its actual type. Here is the setup script: .. literalinclude:: module4_setup.py :linenos: And the generated C code is also included for your information: .. literalinclude:: module4.h :language: c :linenos: .. literalinclude:: module4.c :language: c :linenos: And a testing session is invoked to illustrate the result:: ylan@montop:~/expy/Doc/build/lib.linux-i686-2.5$ python >>> from module4 import * >>> a = counter('VA', 20) >>> b = counter('MD', 14) >>> c = a+b >>> c.name, c.num ('VA&MD', 34) Defining numeric extension of Category 2 is similar, only that the second operand will be of PyObject type, so the prototype would be something like this:: def __add__(me, u): """The C prototype would be: static PyObject* counter___add__(counter* me, PyObject* u);""" And before you work on the argument 'u', you usually do type checking to ensure that it is of the right type. As there are quite a number of numeric methods, enlisting all of them here would be boring. Please refer to the cxpy.py source code, where you shall find all of them. .. _mapping: Create Mapping Types --------------------- You can also implement the mapping protocol for your extension types. Below is a simple example, where our mapping type only help access some static integer array. .. literalinclude:: module5.py :linenos: Note that the three special method names and the signature, which are different from the way you define a pure python mapping type. This is to avoid name conflicts with the sequence protocal (next). And here is the generated C code: .. literalinclude:: module5.h :language: c :linenos: .. literalinclude:: module5.c :language: c :linenos: The testing of this is left for the reader. Again, for a full list of all related special methods, you are encouraged to take a look at the cxpy.py source file. .. _sequence: Create Sequence Types ---------------------- To implement the sequence protocol, simply define those special methods for the sequence protocol. .. literalinclude:: module6.py :linenos: Again notice the choice of the special method names __concat__ and __repeat__, which are not standard in pure python, this is again to avoid name conflict between the special methods for maps and sequences. The generated C code: .. literalinclude:: module6.h :language: c :linenos: .. literalinclude:: module6.c :language: c :linenos: And we have a test session for this module:: ylan@montop:~/expy/Doc/build/lib.linux-i686-2.5$ python Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from module6 import * >>> x = strbuf(100) >>> x += 'OK!' >>> x*= 5 >>> x.buff 'OK!OK!OK!OK!OK!' .. _iterator: Create Iterator Types ---------------------- We will present a relatively complete example, which will provide utilities to test and produce prime numbers, and will factorize an integer via the *iterator protocol*. .. literalinclude:: ../examples/prime.py :linenos: In the prime.py, the prologue actually defines some static storage for the primes. This small module is quite useful, it provides the first 5000 primes, and can test any integers within the range of [1, 2^31-1]. And finally, For any integer in that range, it provides the prime factors by the __iter__ and __next__ methods (iterator protocol). Here is the script file to test this module: .. literalinclude:: ../examples/testprime.py :linenos: The output from this test file looks like this:: ylan@montop:~/expy/examples$ python testprime.py 5000 48611 enumerate prime factors in low to hi order stored primes: 5000 0 0 ; 1 0 ; 2 1 ; 3 1 ; 4 0 ; 5 1 ; 6 0 ; 7 1 ; 8 0 ; 9 0 ; there are 1229 primes from 0 to 9999. The 10th prime is 31 The prime factors for 52 are: [2, 2, 13] Property rem: 1 Property rem modified: 123 [3, 41] And here is the generated C file: .. literalinclude:: ../examples/prime.h :language: c :linenos: .. literalinclude:: ../examples/prime.c :language: c :linenos: .. _memindex: Static array dimension ---------------------- For instance fields, the keyword 'dimension' can be used to declare a static multi-dimensional array:: class myclass(public): size = ifield(int, private, dimension='[2]') the code above would define something like this for the generated C code:: typedef struct { int[2] size; ... }myclass; Of course, you can define multi-dimensions, such as:: dimension='[3][2]' Usually such a field will be 'private', as there ususually not a readily defined way in CPython to access it, but you can define getters/setters for them. .. _exposure: Exposing nested C struct ------------------------ There is another useful keyword in ifield constructor: exposure. Use this field if your extension object has a nested struct and you would like to access it as a member field. Suppose you have a C struct defined as follows:: struct mystruct{ int x, y; }; and you would like to have that struct as a member of your extention object, then expose the field 'x' of it. Here is how to make it happend:: class myclass(public): point = ifield(rawtype('struct mystruct'), private) x = ifield(int, exposure='point.x') In this case, the member 'x' will be considered already exist in the member 'point' as 'point.x', so there won't be another top level 'x' member in the C struct of this class, but the field member would be directly exposed to the python object as a a property named 'x'. Another use case for 'exposure' keyword is to expose it as a different type than what's claimed, which is often used with keyword 'dimension':: class myclass(public): chars = ifield(char, dimension='[4]', exposure=int) this time the 'chars' is of a nominal type 'char', but it can also be seen as an 32 bit integer, because of the 'dimension' keyword, so an 'exposure' keyword is used to achieve this. .. _tp_init: Writing your __init__ function ------------------------------- When you write your __init__ method, you can specify your arguments just the same way as you do in writing a python __init__ method, and EXPY will produce a wrapper function to parse the arguments. What's more, you can also throw exceptions (using the @throws decorator). One thing must remember for __init__: return 0 for success, and -1 for failure. .. _INCREF: Warnings on Py_INCREF ---------------------- There are many special methods, such as __iter__, __next__ etc that require returning a python object, and you should increase the reference count. EXPY will check if Py_INCREF is in there, and will give a warning if not. .. _conclusion: Conclusion ---------- Through examples I hope it all looks interesting and exciting. Suggestions, volunteer works, bug reports are all welcome, I am available via email: lanyjie at yahoo dot come, and expy also has its own email server, as indicated on the homepage. This concludes our introduction to the major features of expy-cxpy. Thanks for your interests, and I hope you find this a nice tool for developing python extensions. Happy coding!