Convert String of an int array to a Numpy array of ints

[mdsousa]

Hi

I am sending a Numpy int array over RabbitMQ and it seem to be receiving correctly.

This is the output RabbitMQ's publish that is sent:

>py msg_send_test.py

Output:
 [x] Sent '0,1617731222,1617731222810382600,4,[ 48 251   9 205]'

This is the output on the receiving end from RabbitMQ's consumer:
>py msg_receive_test.py

Output:
 [*] Waiting for messages. To exit press CTRL+C
[x] message      '0,1617732445,1617732445003573700,4,[209 227 205  72]'
[x] message type <class 'str'>
[x] parts        ['0', '1617732445', '1617732445003573700', '4', '[209 227 205  72]']
[x] parts len    5
[x] imageStr     [209 227 205  72]
[x] imageStr type <class 'str'>

This is the section of code to produce the output from the RabbitMQ consumer:

    message = body.decode()
    parts = message.split(',')
    print("[x] message      %r" % message)
    print("[x] message type %s" % type(message))
    print("[x] parts        %s" % parts)
    print("[x] parts len    %s" % len(parts))
    imageStr = str(parts[4])
    print("[x] imageStr     %s" % imageStr)
    print("[x] imageStr type %s" % type(imageStr))

What I would like to do is convert the imageStr into a Numpy array of ints. The whole array is a string, not an array of individual strings.

Any suggestions would be appreciated.

Thanks...

When I try using Numpy's fromstring method:

    imageArr = np.fromstring(imageStr, dtype=float, sep=' ')
    print("[x] imageArr type %s" % type(imageArr))
    print("[x] ", imageArr)

it gives the following output:

Output:
[x] imageStr type <class 'str'>
[x] imageArr type <class 'numpy.ndarray'>
[x]  []

[mdsousa]

I'm trying this out. In the output from fromstring, I added the following

    imageStr = str(parts[4])
    imageStr = imageStr.replace('[','')
    imageStr = imageStr.replace(']','')
    imageArr = np.fromstring(imageStr, dtype=int, sep=' ')
    print("[x] imageArr type %s" % type(imageArr))
    print("[x] imageArr ", imageArr)

And this is now the output:

Output:
 [*] Waiting for messages. To exit press CTRL+C
[x] message      '0,1617735054,1617735054620256300,4,[104 126 180  87]'
[x] message type <class 'str'>
[x] parts        ['0', '1617735054', '1617735054620256300', '4', '[104 126 180  87]']
[x] parts len    5
[x] imageStr     104 126 180  87
[x] imageStr type <class 'str'>
[x] imageArr type <class 'numpy.ndarray'>
[x] imageArr  [104 126 180  87]

Looks like the '[]' had to be removed from the string, and Numpy puts them back when converting to the int array.

[mdsousa]

I have found that this only works up to some limits.

If the array becomes too large, it only handles the first three elements, I've been able to get it to work up to a 1000 element array. It does not work correctly at 10000 elements. Any idea why the conversion doesn't work?

Thanks...

[mdsousa]

In sending a message (Numpy int array) using RabbitMQ, this is how it is setup and sent

testImage = np.random.randint(0, 255, 10*1000)  # grey scale images
strImage = np.array_str(testImage)
message = <string of meta data> + strImage
channel.basic_publish(exchange='', routing_key='task_queue', body=message, properties=pika.BasicProperties(delivery_mode=2,  # make message persistent
    ))

This is the output of the message sent:

Output:
[x] Message params '0,1617801844,1617801844400951300,10000,[113  75  36 ... 167 157 131]'

In modifying the received message (from rabbitMQ) to get a Numpy int array I've tried:

    message = body.decode()
    parts = message.split(',')
    print("[x] message      %r" % message)
    print("[x] message type %s" % type(message))
    print("[x] parts        %s" % parts)
    print("[x] parts len    %s" % len(parts))
    

This is the output for the above code:

Output:
[x] message      '0,1617801844,1617801844400951300,10000,[113  75  36 ... 167 157 131]'
[x] message type <class 'str'>
[x] parts        ['0', '1617801844', '1617801844400951300', '10000', '[113  75  36 ... 167 157 131]']
[x] parts len    5

I then start working of the message part that is the Numpy array:

    imagePart = parts[4]
    print()
    print("[x] imagePart ", imagePart)
    print(repr(imagePart))
    print(re.sub("\s+", ",", imagePart.strip()))
    print(type(re.sub("\s+", ",", imagePart.strip())))
    print(np.array(re.sub("\s+", ",", imagePart.strip())))
    print(np.array(re.sub("\s+", ",", imagePart.strip())).size)
    print(type(np.array(re.sub("\s+", ",", imagePart.strip()))))
    print(repr(np.array(re.sub("\s+", ",", imagePart.strip()))))
    print()
    

The output for the above code block is (I'm not sure why it thinks it's of dtype 'U27'):

Output:
[x] imagePart  [113  75  36 ... 167 157 131]
'[113  75  36 ... 167 157 131]'
[113,75,36,...,167,157,131]
<class 'str'>
[113,75,36,...,167,157,131]
1
<class 'numpy.ndarray'>
array('[113,75,36,...,167,157,131]', dtype='<U27')

I then try a different approach:

    npPart = np.asarray(imagePart, dtype=np.str)
    npSplit = np.char.split(npPart)
    npSplitPart = np.array(npSplit, dtype=object)
    print(npSplitPart)
    print(npSplitPart.dtype)
    print("[x] npSplitPart type ", type(npSplitPart))
    print("[x] npSplitPart shape ", npSplitPart.shape)
    print("[x] npSplitPart size ", npSplitPart.size)
    print(repr(npSplitPart))
    print(np.array(npSplitPart))

This is the output for the above code block:

Output:
['[113', '75', '36', '...', '167', '157', '131]']
object
[x] npSplitPart type  <class 'numpy.ndarray'>
[x] npSplitPart shape  ()
[x] npSplitPart size  1
array(list(['[113', '75', '36', '...', '167', '157', '131]']),
      dtype=object)
['[113', '75', '36', '...', '167', '157', '131]']

What I would like is to get back to the original Numpy int array. Any idea what I am doing wrong or missing?

Thanks...

[SheeppOSU]

I think it would be easier to do something like this:

import numpy as np
import json

testImage = np.random.randint(0, 255, 10*1000)
metadata = ""
data = {"image": testImage, "metadata": metadata}
string_data = json.dumps(data)  # a string format of the data

# Receiving end
data = json.loads(string_data)
metadata = data["metadata"]
image = data["image"]

[mdsousa]

Thanks SheeppOSU,

I had tried something like that (if '{}' create a list as that is what I did. I read that lists were serializable out of the box.

#sending
listImage = testImage.tolist()

msgList = [imageID, timeStampSeconds, timeStampNano, int(testImage.size)]
msgList += listImage

json.dumps(msgList)

#receiving
message = json.loads(body)
recvBody = list(message)
arr = np.array(msgList[4:len(msgList)]) # metadata is len 4

Thanks again...