TypeError when reading from FIFO

[ralf]

Hi,

I have a strange problem when reading from a named pipe.
When I write into the pipe using the echo command of my shell (zsh) everything works as expected.
When writing into it from a Python script using Pythons write method, I get a TypeError at the readers side of the FIFO.

The following methods are on the readers side:

# ...
pipe = NamedPipe("/tmp/fifo")
pipe.Create()

while True:
    line = pipe.ReadLine()

    if line:
        print(line)
    time.sleep(.1)  # Avoid high CPU load

# ...

class NamedPipe(object):
    # ...
    def Create(self):
        try:
            os.mkfifo(self.path, mode=0o666)
            os.chmod(self.path, 0o666)          # mkfifos mode-argument seems to be ignored.
        except OSError as e:
            if e.errno != errno.EEXIST:
                raise


    def ReadLine(self):

        def opener(path, flags):
            return os.open(path, os.O_RDONLY | os.O_NONBLOCK)

        with open(self.path, buffering=1, opener=opener) as fifo:
            try:
                line = fifo.read()
            except OSError as e:
                if e.errno != errno.EAGAIN or e.errno != errno.EWOULDBLOCK:
                    logging.error("Reading FIFO failed with exception \"%s\"!", str(e))
                line = None
            except Exception as e:
                logging.exception("Reading FIFO failed with exception \"%s\"!", str(e))
                line = None

        if line:
            line = line.rstrip()    # remove trailing \n
        return line

The reader is listening on the named pipe non-blocking.


Lets say, the fifo is at /tmp/fifo, then echo "test" > /tmp/fifo does always work as expected.

But when I write to the pipe from a python script like this:

with open("/tmp/fifo", "w") as fifo:
    fifo.write("test\n")

I get always the following exception on readers side, at "line = fifo.read()"

Error:
Traceback (most recent call last):
  File "/srv/musicdb/lib/namedpipe.py", line 90, in ReadLine
    line = fifo.read()
  File "/usr/lib/python3.6/codecs.py", line 320, in decode
    data = self.buffer + input
TypeError: can't concat NoneType to bytes

Why does echo work, and python not?

---

Somehow the problem is the write-method of python.

Using the following code works:

line="test\n"
fd = os.open("/tmp/fifo", os.O_WRONLY)
os.write(fd, line.encode())
os.close(fd)

Using only a custom opener with "fd = os.open(self.path, os.O_WRONLY)" is not enougth.
The os.write was the solution to my problem.

Does anyone know why the high level I/O does not work?
Are FIFOs so special, or is it a bug in Python?

[Gribouillis]

It looks like a bug in python(!). If you send a complete reader's side example, I can try it here. Obviously the .buffer member in class codecs.BufferedIncrementalDecoder should never become None. See the class here.

For a more elaborate diagnosis, you could run the following file at the beginning of the reader's program

import codecs
from codecs import IncrementalDecoder

class BufferedIncrementalDecoder(IncrementalDecoder):
    """
    This subclass of IncrementalDecoder can be used as the baseclass for an
    incremental decoder if the decoder must be able to handle incomplete
    byte sequences.
    """
    def __init__(self, errors='strict'):
        IncrementalDecoder.__init__(self, errors)
        # undecoded input that is kept between calls to decode()
        self.buffer = b""

    def _buffer_decode(self, input, errors, final):
        # Overwrite this method in subclasses: It must decode input
        # and return an (output, length consumed) tuple
        raise NotImplementedError

    def decode(self, input, final=False):
        # decode input (taking the buffer into account)
        data = self.buffer + input
        (result, consumed) = self._buffer_decode(data, self.errors, final)
        # keep undecoded input until the next call
        self.buffer = data[consumed:]
        return result

    def reset(self):
        IncrementalDecoder.reset(self)
        self.buffer = b""

    def getstate(self):
        # additional state info is always 0
        return (self.buffer, 0)

    def setstate(self, state):
        # ignore additional state info
        self.buffer = state[0]


# Replace the class in module codecs by our own class
codecs.BufferedIncrementalDecoder = BufferedIncrementalDecoder

Then the codecs module will use your own version of the class. You can now tweak the class' code to detect exactly which bytes are read, when does the buffer becomes None, etc. For example you could turn .buffer into a property and raise an exception as soon as it receives the value None.