Python Forum
How can I make this code more efficient and process faster?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How can I make this code more efficient and process faster?
#1
The main function simulates receiving as much data as possible and then forwarding it to other subprocesses for processing. There will actually be multiple subprocesses, and only one subprocess is tested here. The data calculation in the subprocess cannot affect the data reception, and the calculation time is uncertain and may be longer. Below is my test code. I want to know how to optimize it to make it faster.
import multiprocessing
import threading
import collections
import time

def doCount(data, num):
    num=num+1
    #print("doCount:",time.time_ns())
    time.sleep(0.000001)

def threadWorker(d,event):
    tstart = time.time_ns()
    num = 0
    while True:
        if d:
            data = d.popleft()
            if data is None:
                break
            doCount(data,num)
        else:
            event.clear()
            event.wait()

    tend = time.time_ns()
    print('thread-recv:', (tend - tstart) / 1000000)

def processWorker(conn):
    d = collections.deque()
    event = threading.Event()
    t = threading.Thread(target=threadWorker, args=(d,event), daemon=True)
    t.start()

    cstart = time.time_ns()
    while True:
        try:
            data = conn.recv()
            d.append(data)
            event.set()
            if data is None:
                break
        except EOFError:
            break

    cend = time.time_ns()
    print('recv:',(cend- cstart)/1000000)
    print('thread-wait:', time.time_ns())
    t.join()
    print('thread-end:', time.time_ns())

def main():
    read_conn, write_conn = multiprocessing.Pipe(duplex=False)

    p = multiprocessing.Process(target=processWorker, args=(read_conn,))
    p.start()

    pstart = time.time_ns()
    for i in range(100000):
        write_conn.send(
            {"code": f"code {i}", "time": time.time_ns(), "type": 1, "num": 100000, "aaa": "ddd", "bbbb": "dddd",
             "ccc": "dddd", "fff": "dddd", "ggg": "dddd", "hhh": "dddd", "iii": "dddd"})

    write_conn.send(None)
    write_conn.close()

    pend = time.time_ns()
    print('send:', (pend - pstart) / 1000000)

    p.join()

if __name__ == '__main__':
    main()
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
Question I'm trying to make a piece of code Alislugacool 5 3,333 Sep-13-2025, 06:11 AM
Last Post: MilesWeb
  Make code run faster: point within polygon lookups Bennygib 3 1,475 Jul-11-2025, 07:24 AM
Last Post: FrankBuckland
  Cleaning my code to make it more efficient BSDevo 14 5,635 Jul-11-2025, 07:20 AM
Last Post: FrankBuckland
  hi need help to make this code work correctly atulkul1985 5 2,843 Nov-20-2023, 04:38 PM
Last Post: deanhystad
  newbie question - can't make code work tronic72 2 2,162 Oct-22-2023, 09:08 PM
Last Post: tronic72
  A more efficient code titanif 2 1,797 Oct-17-2023, 02:07 PM
Last Post: deanhystad
  how to make bot that sends instagram auto password reset code kraixx 2 3,404 Mar-04-2023, 09:59 PM
Last Post: jefsummers
  Make code non-blocking? Extra 0 2,544 Dec-03-2022, 10:07 PM
Last Post: Extra
  faster code for my code kucingkembar 19 8,706 Aug-09-2022, 09:48 AM
Last Post: DPaul
  How to make app run only when process is aviable Mawixy 1 2,081 Apr-19-2022, 03:45 PM
Last Post: Axel_Erfurt

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020