Skip to content
85 changes: 85 additions & 0 deletions Doc/extending/embedding.rst
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,91 @@ will depend on the details of the C++ system used; in general you will need to
write the main program in C++, and use the C++ compiler to compile and link your
program. There is no need to recompile Python itself using C++.

.. _freezingandembeddingmanually:

Freezing Modules in Manually Embedded Python
============================================

It is possible to use frozen modules in embedded Python if the main module
is also frozen. It seems people are bitten when they don't have their main module

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by 'bitten'? I think it's better to use direct language and be clear on what the problem might be.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As in when they try to run it, it won't work properly.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a specific exception/error that appears, or does it do something like hang?

@AraHaan AraHaan Oct 23, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it crashes or errors.

frozen and use ``PyRun_SimpleString`` that contains their main module(s) but
also depend on an frozen module (like the ``__hello__`` module). However if we
wanted to add our own module to the frozen list we would normally do this::

#include <importlib.h>
#include <importlib_external.h>
#include "mymodule.h"

static const struct _frozen _PyImport_FrozenModules[] = {
/* importlib */
{"_frozen_importlib", _Py_M__importlib, (int)sizeof(_Py_M__importlib)},
{"_frozen_importlib_external", _Py_M__importlib_external,
(int)sizeof(_Py_M__importlib_external)},
/* mymodule */
{"mymodule", M_mymodule, (int)sizeof(M_mymodule)},
{0, 0, 0} /* sentinel */
};

const struct _frozen * PyImport_FrozenModules = _PyImport_FrozenModules;

.. note::
The reason why this example includes ``importlib.h`` and
``importlib_external.h`` is because of the fact that the embedded
interpreter might need them to run properly (load the Python standard
library from an zip file or the sources like normally). Also in order
for your compiler to work you must point it to the Python subdirectory
in an active clone of the cpython repository or source tarball.
However it is not required to manually build the python core if you are
using an clone or source tarball of an version installed on your system.

As you can see the above code will compile (with an warning on Windows
which will export ``PyImport_FrozenModules`` on the embedded python program).
This is not what we want. And if we were to run it with ``myprogram`` we will
get the following traceback on running the string-based main module(s).

.. code-block:: py

Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'mymodule'

There is got to be a better way to have what we want but to also make python
aware of our ``mymodule`` being an frozen module. This module could be
anything from an import hook or anything else where you want to freeze
it similar to how ``importlib`` is frozen to support how your code base
currently is.

The fix to this is to change the C code above to::

#include <importlib.h>
#include <importlib_external.h>
#include "mymodule.h"

static const struct _frozen _PyImport_FrozenModules[] = {
/* importlib */
{"_frozen_importlib", _Py_M__importlib, (int)sizeof(_Py_M__importlib)},
{"_frozen_importlib_external", _Py_M__importlib_external,
(int)sizeof(_Py_M__importlib_external)},
/* mymodule */
{"mymodule", M_mymodule, (int)sizeof(M_mymodule)},
{0, 0, 0} /* sentinel */
};

And then in the main() C or C++ function in your embedded interpreter add this line::

PyImport_FrozenModules = _PyImport_FrozenModules;

Now your embedded Python should be able to load your frozen modules perfectly fine.

.. note::
This logic was borrowed from Programs/_freeze_importlib.c. Also using this method is
the only way to use frozen modules in Windows as well. On further note
Programs/_freeze_importlib.c in the cpython source tree (with minimal changes) could
be changed to act like the normal python freeze tool as the normal freeze tool
currently does not work on Windows so it makes it only possible to freeze to an
header file, make the frozen module list manually, write the embedded python
entry manually and then compile to have an fully featured embedded python interpreter
with your own frozen modules.

.. _compiling:

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Added more documentation on manually embedding python with custom frozen
modules.