Calling python from another program

[eedjsa]

Hi,

1. I have a simple python script named script.py:

expr = input()      # read an expression from stdin
eval(expr)          # evaluate the expression

2. I start this script from a C/C++ program main.cc:

#include <string.h>
#include <unistd.h>
#include <sys/types.h>          /* See NOTES */
#include <sys/socket.h>

#include <cstdio>
#include <iostream>

using namespace std;

int
main()
{
int spair[2];
   socketpair(AF_UNIX, SOCK_STREAM, 0, spair);
const int & parent_sock = spair[0];
const int & python_sock = spair[1];

   if (fork())   // parent process (main.cc)
      {
         close(python_sock);   // not used by parent

         const char * expr = "print('Hello')\n";   // python expression
         write(parent_sock, expr, strlen(expr));   // write it to python's stdin
         char buffer[100];
         int len = read(parent_sock, buffer, sizeof(buffer));   // python output
         buffer[len] = 0;

         cout << buffer << endl;
      }
   else          // child process (python3)
      {
         close(parent_sock);   // not used by child
         close(STDIN_FILENO);
         close(STDOUT_FILENO);
         close(STDERR_FILENO);
         dup(python_sock);   // stdin
         dup(python_sock);   // stdout
         dup(python_sock);   // stderr

         char * program = const_cast<char *>( "/usr/bin/python3"       );
         char * script  = const_cast<char *>( "./script.py" );
         char * argv[] = { program, script, 0 };
         char * env[] = { 0 };
         execve(argv[0], argv, env);
      }
}

3. Not surprisingly, compiling and running main prints Hello:

$ ./main
Hello

4. However, if I remove script.py from argv:

         char * argv[] = { program, 0 };

then the result should be the same, shouldn't it? The python interpreter should
read print('Hello') , execute it, and print Hello. But what happens instead is
that the program main hangs.

I have checked the file descriptors with lsof - nothing strange there.
I have checked the hanging main with strace:

strace: Process 16712 attached
read(0,

So it seems like /usr/bin/python3 blocks on reading stdin (as expected),
but the write() from main does not get there. Even though (as the previous
test (3. above) with script.py has proven) the connection from main to
/usr/bin/python3 works properly.

Can someone please explain what is going on here? My platform is Linux Mint (Ubuntu),
compiler g++, Python 3.10.12.

Thanks,
Jürgen

[Gribouillis]

Can someone please explain what is going on here?

I'm not able to do that. Some difference between Python's interactive mode and non-interactive mode, perhaps in the handling of the communication pipes.

That said, if you want to execute Python code from the C++ program, why not using the Python C API? This example from the official documentation should allow you to run the hello world command and many others.

[bowlofred]

I suspect the difference is that your intermediate script uses input() to read (and process) a single line. But when not using the script, the python process is waiting for an EOF to signal the end of the program. But that doesn't arrive.

[bowlofred]

In particular, when I strace it, I see that python has read the program via stdin, it just is waiting for more input.

Output:
...
31803 read(0, "print('Hello')\n", 4096) = 15
31803 read(0,  <unfinished ...>
...

If you attach strace to it after this point, you'll just see it wating for more input and will have missed that the program has already arrived.

[bowlofred]

Shutting down the socket after you've sent the data works here.

#include <string.h>
#include <unistd.h>
#include <sys/types.h>          /* See NOTES */
#include <sys/socket.h>

#include <cstdio>
#include <iostream>

using namespace std;

int
main()
{
int spair[2];
   socketpair(AF_UNIX, SOCK_STREAM, 0, spair);
const int & parent_sock = spair[0];
const int & python_sock = spair[1];

   if (fork())   // parent process (main.cc)
      {
         close(python_sock);   // not used by parent

         const char * expr = "print('Hello')\n";   // python expression
         write(parent_sock, expr, strlen(expr));   // write it to python's stdin
         shutdown(parent_sock, SHUT_WR);
         char buffer[100];
         int len = read(parent_sock, buffer, sizeof(buffer));   // python output
         buffer[len] = 0;

         cout << buffer;
      }
   else          // child process (python3)
      {
         close(parent_sock);   // not used by child
         close(STDIN_FILENO);
         close(STDOUT_FILENO);
         close(STDERR_FILENO);
         dup(python_sock);   // stdin
         dup(python_sock);   // stdout
         dup(python_sock);   // stderr

         char * program = const_cast<char *>( "/usr/bin/python3"       );
         // char * script  = const_cast<char *>( "./script.py" );
         char * argv[] = { program, 0 };
         char * env[] = { 0 };
         execve(argv[0], argv, env);
      }
}

[eedjsa]

Can someone please explain what is going on here?

I'm not able to do that. Some difference between Python's interactive mode and non-interactive mode, perhaps in the handling of the communication pipes.

That said, if you want to execute Python code from the C++ program, why not using the Python C API? This example from the official documentation should allow you to run the hello world command and many others.

Hi,
interesting idea, thanks. I wasn't aware of that API. However, the C++ snippet that I posted was only a small piece of a huge C++ program and the this
piece was an optional part (i.e. the huge program should be able to run even if python or the python modules that it uses were not installed. Therefore
linking with anything was not an option.

[eedjsa]

Shutting down the socket after you've sent the data works here.

#include <string.h>
#include <unistd.h>
#include <sys/types.h>          /* See NOTES */
#include <sys/socket.h>

#include <cstdio>
#include <iostream>

using namespace std;

int
main()
{
int spair[2];
   socketpair(AF_UNIX, SOCK_STREAM, 0, spair);
const int & parent_sock = spair[0];
const int & python_sock = spair[1];

   if (fork())   // parent process (main.cc)
      {
         close(python_sock);   // not used by parent

         const char * expr = "print('Hello')\n";   // python expression
         write(parent_sock, expr, strlen(expr));   // write it to python's stdin
         shutdown(parent_sock, SHUT_WR);
         char buffer[100];
         int len = read(parent_sock, buffer, sizeof(buffer));   // python output
         buffer[len] = 0;

         cout << buffer;
      }
   else          // child process (python3)
      {
         close(parent_sock);   // not used by child
         close(STDIN_FILENO);
         close(STDOUT_FILENO);
         close(STDERR_FILENO);
         dup(python_sock);   // stdin
         dup(python_sock);   // stdout
         dup(python_sock);   // stderr

         char * program = const_cast<char *>( "/usr/bin/python3"       );
         // char * script  = const_cast<char *>( "./script.py" );
         char * argv[] = { program, 0 };
         char * env[] = { 0 };
         execve(argv[0], argv, env);
      }
}

[eedjsa]

Ahh, shutdown. That shines some light on what is happening. Even though I believe that in interactive mode python will not shutdown its stdin.
My plan was to work with C++ and python in a ping-pong fashion (do something in C++, then something in pytho, then something in C++, and
so on). But this gives me a way to work around the problem. Thanks a lot.

[Gribouillis]

Therefore
linking with anything was not an option.

Instead of forking, and execve to execute /usr/bin/python3 in the child process, you could execve to execute another C++ program that uses the Python C API and communicates with the parent process. This way the main C++ program doesn't need to link against the Python library.

[eedjsa]

Therefore
linking with anything was not an option.

Instead of forking, and execve to execute /usr/bin/python3 in the child process, you could execve to execute another C++ program that uses the Python C API and communicates with the parent process. This way the main C++ program doesn't need to link against the Python library.

I see, thanks. I now have more options than problems. My "big" program is open source and I want to make its installation as simple as possible
for the users. Since I distribute my program in source, the users would need to install the python C API (which creates one more dependency),
right? My favored solution so far is execve with a separate python script (which has grown longer than I initially thought it would because it has
to do a few things that are simpler in python than in C++).