unicode and utf-8

[sorynturda]

Hey guys!
I don't have much experience with Python and I need some clarifications.

What I want to do?
I want to keep the some data in Unicode format. The data is sent to a third-party and stored there.

From what I've read "Python often defaults to using it" (https://docs.python.org/3/howto/unicode.html)

Given the use case:
I recieve the data from multiple databases, some data is recieved as bytes some as string(str).
I do that:

data = recieved_data.decode("utf-8").

Is there anything to do to convert the recieved data to Unicode?

Thanks alot!
Sorin

[Gribouillis]

Is there anything to do to convert the recieved data to Unicode?

The line that you wrote converts the data to a Python unicode string, provided the data was previously encoded with the utf8 encoding in the database. Unicode is not a format, unicode is a universal alphabet that contains all the letters of all the languages in the world and more. A unicode string is a sequence of symbols from this gigantic alphabet. If you want to store unicode data into a file, it must be encoded in some way.

So what does your question mean exactly?

I must add that in order to send unicode data through a network, it must also be encoded, but this is often hidden from the client code by the software components that implement network communications.

[sorynturda]

I get the data from mongodb using pymongo, where it is stored. Then, in the code, is converted to a dict of strings. ONLY str data type.
After that, the data is processed and sent to a third-party, which is external and the data sent must be in unicode format.

And here I have doubts. Does the data need to processed in order to be in unicode format or it is already in unicode format.
Same goes for other data type, that comes from ldap but as bytes. I just decode it to process it (add some elements in list, remove some, et cetera)

I must add that in order to send unicode data through a network, it must also be encoded, but this is often hidden from the client code by the software components that implement network communications.

Yes, basically this is what it happens in code. Data is sent via HTTP or TCP (LDAP) to third-parties.

[Gribouillis]

Does the data need to processed in order to be in unicode format or it is already in unicode format.

Again, UNICODE IS NOT A FORMAT, there is no way to format anything in "unicode format". UTF-8 or UTF-16 are formats to encode unicode characters. ISO-8859-1 or euc_jisx0213 are encoding formats that encode some ranges of unicode characters.

Starting with Python 3.0, the Python type str is an implementation of abstract unicode strings. Conceptually, a str instance is a sequence of unicode code points. If you want to write it to a file or send it through a socket, you need to choose an encoding because files and sockets don't accept abstract unicode characters, they accept only bytes (but again this may be hidden by software components).

[DeaD_EyE]

Watch some YouTube videos about Unicode and utf8.

Actually, CPython stores internally str in utf8, but this is an implementation detail.


If you work with text files, the encoding is mostly utf8. Sometimes there are old text files with a complete different encoding.