Popen - How can I read data before CTRL+C command is issued

[maffaz]

I have a command line tool in windows that scans for devices.

It will loop through all devices in range repeating until it receives a CTRL+C command.

device1 name firmware
device2 name firmware
device3 name firmware
device1 name firmware
device2 name firmware
device3 name firmware
...

I want to stop the scan when it reaches a certain device so I can issue an update firmware command.

At the moment I can only capture the output after the CTRL+C command is issued using the following function that starts the scan, sleeps, then issues the CTRL+C command, I then catch the error and process the output in the except block:

            command = [self.cli_tool, '-s']

            startupinfo = subprocess.STARTUPINFO()
            startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
            startupinfo.wShowWindow = subprocess.SW_HIDE

            stream = []

            if self.check_dongle_firmware() is not False:
                try:                    
                    self.proc = subprocess.Popen(
                                command, 
                                stdin=subprocess.PIPE, 
                                stdout=subprocess.PIPE, 
                                stderr=subprocess.DEVNULL, 
                                startupinfo=startupinfo)

                    time.sleep(SCAN_TIMEOUT)
                    os.kill(self.proc.pid, signal.CTRL_C_EVENT)
                    self.proc.wait()

                except KeyboardInterrupt:                
                    for line in self.proc.stdout:
                        stream.append(line)							

                    for x in stream[7:]:
                        x = x.decode()
                        print(x.strip())   

I want something like this:

    stream = []

     self.proc = subprocess.Popen(
                                command, stdin=subprocess.PIPE,
                                stdout=subprocess.PIPE, 
                                stderr=subprocess.DEVNULL, 
                                startupinfo=startupinfo)

    for line in self.proc.stdout:
        stream.append(line)	
		if 'device' in line:
			os.kill(self.proc.pid, signal.CTRL_C_EVENT)
			self.proc.wait()

This does not work though.

I tried various combinations using :

universal_newlines=True

bufsize=1

self.proc.communicate()[0]

When I try the code using a command like ping I have no problem and full control of the output.

I have searched SO for tried anything that looks similar but nothing works for what I need.

I think I am missing something obvious or it is not possible due to the tool locking or being in a separate process not accessible by python.

Any direction or advice is appreciated!

[Larz60+]

exception KeyboardInterrupt

Raised when the user hits the interrupt key (normally Control-C or Delete). During execution, a check for interrupts is made regularly. The exception inherits from BaseException so as to not be accidentally caught by code that catches Exception and thus prevent the interpreter from exiting.

This definition states that it will unconditionally exit.
Nevertheless, this write up seems to indicate an 'unreliable' work around: https://stackoverflow.com/questions/4606...-in-python

examples found here: http://nullege.com/codes/search?cq=os.kill seem to follow the same format as the one you are using

[maffaz]

I can handle the Exception no problem.

The problem I have is that I want to issue the CTRL+C command upon hitting a certain device.

I cannot read the data until the CTRL+C command is sent and the tool exits..

I want to read the data and issue a CTRL+C command if that data is the data I require. in this instance 'device2'.

so my ideal solution is:

1. start command.

2. read each line and append to a list.

3. after each line is appended, check the list for 'device2'

4. if 'device2' is in that list, issue a CTRL+C command.

5. then I can assign that line to a variable and use it to update the firmware.

[Larz60+]

can you do it if a variable?
ctrlc = chr(5)
then issue ctrlc?

[volcano63]

You can can try reading from self.proc.stdout till you get your device in the output - and then kill the process.

for line in self.proc.stdout:
    if <> in line:
        self.proc.kill()

(I think).

The problem is - it may get stuck. And what happens if the device never shows up?

I used to write stdout to a file stdout=open(<log name>, 'w') - and then periodically look-up the file content in a loop, and leaving the loop after some maximum number of attempts.

[maffaz]

You can can try reading from self.proc.stdout till you get your device in the output - and then kill the process.

for line in self.proc.stdout:
    if <> in line:
        self.proc.kill()

(I think).

The problem is - it may get stuck. And what happens if the device never shows up?

I used to write stdout to a file stdout=open(<log name>, 'w') - and then periodically look-up the file content in a loop, and leaving the loop after some maximum number of attempts.

I tried that but I cannot actually read the stream in the first place.

I also tried saving to a file.

In both cases, It will save to a file/list no problem but only AFTER the CTRL+C command is pressed.

kill etc does not allow save anything!

I can only capture the data after the CTRL+C is issued. Before that I get nothing.

It is buffered data and seems to be in memory until CTRL+C and then it spits out what is there!

---

can you do it if a variable?
ctrlc = chr(5)
then issue ctrlc?

Yes, it is also possible but This is not the issue.

I need to read the data and save to a variable before the CTRL+C command is issued

[volcano63]

Cannot be done, you say ? This was done on Ubuntu

Output:
In [23]: ping_proc = subprocess.Popen(['ping', 'google.com'], 
    ...:                              stdout=subprocess.PIPE, stdin=subprocess.PIPE, 
    ...:                              stderr=subprocess.STDOUT, universal_newlines=True)

In [24]: for line_no, line in enumerate(ping_proc.stdout):
    ...:     print(line,)
    ...:     if line_no == 10:
    ...:         print('Killing process')
    ...:         ping_proc.kill()
    ...:         break
    ...:     
64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=3 ttl=50 time=66.0 ms

64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=4 ttl=50 time=66.1 ms

64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=5 ttl=50 time=66.3 ms

64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=6 ttl=50 time=66.0 ms

64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=7 ttl=50 time=66.2 ms

64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=8 ttl=50 time=66.1 ms

64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=9 ttl=50 time=66.1 ms

64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=10 ttl=50 time=66.2 ms

Killing process

I redirect stderr to stdout - some processes use stderr instead of stdout, so suppressing the former is a bad idea

[maffaz]

Cannot be done, you say ? This was done on Ubuntu

Output:
In [23]: ping_proc = subprocess.Popen(['ping', 'google.com'], 
    ...:                              stdout=subprocess.PIPE, stdin=subprocess.PIPE, 
    ...:                              stderr=subprocess.STDOUT, universal_newlines=True)

In [24]: for line_no, line in enumerate(ping_proc.stdout):
    ...:     print(line,)
    ...:     if line_no == 10:
    ...:         print('Killing process')
    ...:         ping_proc.kill()
    ...:         break
    ...:     
64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=3 ttl=50 time=66.0 ms

64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=4 ttl=50 time=66.1 ms

64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=5 ttl=50 time=66.3 ms

64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=6 ttl=50 time=66.0 ms

64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=7 ttl=50 time=66.2 ms

64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=8 ttl=50 time=66.1 ms

64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=9 ttl=50 time=66.1 ms

64 bytes from lhr25s10-in-f14.1e100.net (216.58.198.174): icmp_seq=10 ttl=50 time=66.2 ms

Killing process

I redirect stderr to stdout - some processes use stderr instead of stdout, so suppressing the former is a bad idea

Interesting! That is golden advice even if it does not work..

The ping example works fine with stdout but not the tool in question.

I am not at my computer until this evening and I will try it then but that is certainly something I have not tried as of yet!

I must stress that it is a windows CLI Tool but the beauty of subprocess is that it is cross platform.

Is there anything to consider if its windows?

[volcano63]

Interesting! That is golden advice even if it does not work..

The ping example works fine with stdout but not the tool in question.

I am not at my computer until this evening and I will try it then but that is certainly something I have not tried as of yet!

I must stress that it is a windows CLI Tool but the beauty of subprocess is that it is cross platform.

Is there anything to consider if its windows?

Did you try to set stderr=subprocess.STDOUT? Some apps send output to stderr - don't ask me why, but I saw it happening, especially with HW-oriented apps.

One more thing I have forgotten (I haven't used subprocess for several months) - by default its output is bytestrings, universal_newlines=True forces its output to unicode strings.

---

If you run your command in terminal, does it provide continuous output - or only after you stop it? In the former case, my approach should work

[maffaz]

Interesting! That is golden advice even if it does not work..

The ping example works fine with stdout but not the tool in question.

I am not at my computer until this evening and I will try it then but that is certainly something I have not tried as of yet!

I must stress that it is a windows CLI Tool but the beauty of subprocess is that it is cross platform.

Is there anything to consider if its windows?

Did you try to set stderr=subprocess.STDOUT? Some apps send output to stderr - don't ask me why, but I saw it happening, especially with HW-oriented apps.

One more thing I have forgotten (I haven't used subprocess for several months) - by default its output is bytestrings, universal_newlines=True forces its output to unicode strings.

---

If you run your command in terminal, does it provide continuous output - or only after you stop it? In the former case, my approach should work

If i run it in a command it will print the output to the screen until CTRL+C is pressed. Then it will exit and the data remains.

It is a HW orientated app. It is a dongle that communicates with a Temperature sensor written in C using a Serial(com) port via USB.

There can be many in range and i am writing a program that will iterate through each one and update the firmware if necessary.

The program works fine except for only being able to obtain the data by using a timer and reviewing the output presented after it closes.

I will try this evening when I return home as I have not yet tried your suggestion.

Indeed where I am, it works fine with ping but also with stdout.