Skip to content

Memory Consumption Problem in 2.1.0 #147

@SegFaulty

Description

@SegFaulty

after upgrading today to 2.1.0 (on ubuntu 16.04)
We noticed a permanently increasing memory consumption in the run.
This resulted in many cases in out of memory errors.
We had to switch back to version 2.0.3 (We tested 2.0.1, this version was fine too)

We are streaming and decoding from a testfile with 600MB and 2Mio Messages.
Version 2.0.1 and 2.0.3 needed 4-6 MB php-memory for streaming through the whole file.
Version 2.1.0 increases memory with every message until its fails at 750MB memory limit!

php -i of failure version:
msgpack
extension Version => 2.1.0
header Version => 3.2.0

We used execute, unpack, reset ... see code below:

protected function getMessages__php_msgpack($messageCount = 1){
	$messages = array();
	if( $this->filePointer ){
		$continue = true;
		while( $continue ){
			$offset = 0;
			$this->unpacker->reset();
			$executeResult = $this->unpacker->execute($this->buffer, $offset);
			// since msgp-version 0.5.8 execute is true even if the msg-pack string is incomplete, but then offset is greater then buffer-size
			if( $executeResult AND $offset<=strlen($this->buffer) ){ 
				$this->buffer = substr($this->buffer, $offset);
				$messages[] = $this->unpacker->data();
				if( count($messages)==$messageCount ){
					$continue = false;
				}
			} else{
				// buffer not decoded -> feed more
				if( !feof($this->filePointer) ){
					// feed the buffer
					$this->buffer .= fread($this->filePointer, $this->readBufferSize);
				} else{
					// end of file
					$continue = false;
					$this->endOfFile = true;
					if( $this->buffer!='' ){
						trigger_error(__METHOD__.' buffer not empty at eof:'.Waps_Helper_String::getStringRepresentationOfBinary($this->getBuffer()), E_USER_WARNING);
						$this->errorOccurred = true;
					}
				}
			}
		}
	}
	return $messages;
}

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions