Multi-processing - problem with running multiple *.py files at the same time

[Antonio]

Hi there,

I have multiple *.py files that I need to run concurrently. Each file looks like this:

def my_function(file):

    # my code

Then I have written this to run my files simultaneously:

import multiprocessing

from file1 import my_function

if __name__ == '__main__':
    __spec__ = "ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>)" # This line is because I am using Spyder as my IDE

    files = ["path/to/file1.py","path/to/file2.py","path/to/file3.py","path/to/file4.py","path/to/file5.py"]

    for item in files:
        p = multiprocessing.Process(target=my_function, args=(item,))
        p.start()
        p.join()
        print("Done!")

The issue is that, however, the files are still being run one after another like in a sequence but not in parallel! Do you have any ideas how I may run my files concurrently?

[woooee]

How do you know they are not run in parallel? Note that your program will print"Done" after every thread is started. To print when they are done, use something like

import multiprocessing
import time

    files = ["path/to/file1.py","path/to/file2.py",
             "path/to/file3.py","path/to/file4.py",
             "path/to/file5.py"]
 
    process_list=[]
    for item in files:
        p = multiprocessing.Process(target=my_function, args=(item,))
        p.start()
        process_list.append(p)

    still_alive=True
    while still_alive:
        print("-"*50)
        still_alive=False
        for p in process_list:
            if p.is_alive():
                still_alive=True
                print(p)
                time.sleep(0.2)
                p.join()

    print("finished") 

[Antonio]

I check the task manager and it's only one core of the CPU that is working; plus the time is 5 times more than running a single file (I have 5 files in the loop as you see). The point is, how do I run the for loop in parallel?

Hi there,

I have multiple *.py files that I need to run concurrently. Each file looks like this:

def my_function(file):

    # my code

Then I have written this to run my files simultaneously:

import multiprocessing

from file1 import my_function

if __name__ == '__main__':
    __spec__ = "ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>)" # This line is because I am using Spyder as my IDE

    files = ["path/to/file1.py","path/to/file2.py","path/to/file3.py","path/to/file4.py","path/to/file5.py"]

    for item in files:
        p = multiprocessing.Process(target=my_function, args=(item,))
        p.start()
        p.join()
        print("Done!")

The issue is that, however, the files are still being run one after another like in a sequence but not in parallel! Do you have any ideas how I may run my files concurrently?

[woooee]

The code I posted, now modified to print each process that is still running, will tell you what is running. Post back the results of the code I posted above.

[Antonio]

The code I posted, now modified to print each process that is still running, will tell you what is running. Post back the results of the code I posted above.

It does not work in parallel and consumes same amount of time. Plus all I get as the final output is:

--------------------------------------------------
finished

[volcano63]

From Multiprocessing - RTM

join([timeout])
If the optional argument timeout is None (the default), the method blocks until the process whose join() method is called terminates. If timeout is a positive number, it blocks at most timeout seconds. Note that the method returns None if its process terminates or if the method times out. Check the process’s exitcode to determine if it terminated.

A process can be joined many times.

A process cannot join itself because this would cause a deadlock. It is an error to attempt to join a process before it has been started.

You essentially block the parent process till the current child process terminates, making all the multiprocessing effort redundant.

You may wait on each process after your start all the processes. Drop p.join() from the first loop and use p.join(0.2) in the second one (also, drop time.sleep from it - redundant).


In general, I believe process pool could be a better option (haven't done multiprocessing in Python for ages) - but in your case that is not so important. If you apply the fixes I've specified above, your code should work as you expect it to.