[solved] Variable number of dictionnaries as argument in def()

[paul18fr]

In the current code, I’m performing tests to works with a variable number of dictionaries; I’m looking for a way to get all keys and their values.

In the current example, especially in DictionnaryTest2 function, I fail in changing the tuple into the original list, in order to get key names (and an error is coming): I’m wondering if I’m using the right way (this is the first time) and so how to proceed?

Thanks for any advice

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import numpy as np

# A dictionnary is created
n = 10
m = 1
X = np.random.random( (n,m) )
Y = np.random.random( (n,m) )
Z = np.random.random( (n,m) )
MyDictionnary = {'Abcissa': X, 'Ordinate': Y, 'Altitude': Z}
MyDictionnary2 = {'Abcissa2': X, 'Ordinate2': Y, 'Altitude2': Z, 'Theta': (X+Y)}
del X, Y, Z

# Dictionnary keys are listed / the dictionnary is explicitly expressed
KeyList0 = list(MyDictionnary.keys())
print("Key list (explicitly) : {}".format(KeyList0))


# Dictionnary keys are listed / the dictionnary name is a variable
MyVar = 'MyDictionnary'
KeyList1 = list(locals()[MyVar].keys())
print("Key list (name=variable) : {}".format(KeyList1))


# Now inside a function with the dictionnary in argument
def DictionnaryTest1(MyDict):
  NewVar = 'MyDict'
  KeyListFunction = list(locals()[NewVar].keys())
  print("Key list in a function : {}".format(KeyListFunction))
  return
  
KeyList1 = DictionnaryTest1(MyDictionnary)



# A list a dictionnary names is now created
DictionnaryNamesTables = ['MyDictionnary', 'MyDictionnary2']

# just for printing the dictionnaries list
for i in range(len(DictionnaryNamesTables)): 
  print(DictionnaryNamesTables[i])
  

def DictionnaryTest2(*args):

  # tests
  print("Type args = {}".format(type(args)))
  print("length args = {}".format(len(args)))
  print("args = {}".format(args))
  args = list(args)
  print("length args (after list())= {}".format(len(args)))
  NumberOfDictionnaries = len(args)
  
  for i in range(len(args)):
    NewVar = args[i]  #
    print("NewVar = {}".format(NewVar))
    KeyListFunction2 = list(locals()[NewVar].keys())
    print("KeyListFunction2 = {}".format(KeyListFunction2))

  return

KeyList2 = DictionnaryTest2(DictionnaryNamesTables)

Output:
Type args = <class 'tuple'>
length args = 1
args = (['MyDictionnary', 'MyDictionnary2'],)
length args (after list())= 1
NewVar = ['MyDictionnary', 'MyDictionnary2']

and the (logical) error

Error:
TypeError: unhashable type: 'list'

[ndc85430]

What are you actually trying to achieve? I'm concerned about all this dynamic code creation you're doing - here using locals. Do you really need to be doing that? Code like that tends to be quite difficult to maintain.

[buran]

There are so many issues here
I think they want to unpack DictionnaryNamesTables in the function call

KeyList2 = DictionnaryTest2(*DictionnaryNamesTables)

Of course then they will hit scope issues because of using locals() - MyDictionnary is not defined inside the function scope.

Output:
Key list (explicitly) : ['Abcissa', 'Ordinate', 'Altitude']
Key list (name=variable) : ['Abcissa', 'Ordinate', 'Altitude']
Key list in a function : ['Abcissa', 'Ordinate', 'Altitude']
MyDictionnary
MyDictionnary2
Type args = <class 'tuple'>
length args = 2
args = ('MyDictionnary', 'MyDictionnary2')
length args (after list())= 2
NewVar = MyDictionnary

Error:
Traceback (most recent call last):
  File "***", line *, in <module>
    KeyList2 = DictionnaryTest2(*DictionnaryNamesTables)
  File "***", line **, in DictionnaryTest2
    KeyListFunction2 = list(locals()[NewVar].keys())
KeyError: 'MyDictionnary'

[paul18fr]

Dictionnaries names and numbers are variables, as well as for their keys ; using locals() is the only way I've found so far to not explicitly enter such informations.

Is it the best good way to do that? probably not, and I'm open to any suggestion to improve it in an efficient way

[buran]

as mentioned by @ndc85430 it is unclear what are you actually trying to achieve

maybe something like this?

from itertools import chain
spam = {'foo':1, 'bar':2}
eggs = {'x':0, 'y':3}

dicts = [spam, eggs]

def get_keys(*args):
    for some_dict in args:
        print(list(some_dict.keys()))

def get_keys2(*args):
    print(list(chain(*args)))

get_keys(*dicts)
get_keys2(*dicts)

Output:
['foo', 'bar']
['x', 'y']
['foo', 'bar', 'x', 'y']

[paul18fr]

At least for such intent, I guess I figured out

Thanks

import numpy as np
from itertools import chain

# A dictionnary is created
n = 10
m = 1
X = np.random.random( (n,m) )
Y = np.random.random( (n,m) )
Z = np.random.random( (n,m) )
MyDictionnary = {'Abcissa': X, 'Ordinate': Y, 'Altitude': Z}
MyDictionnary2 = {'Abcissa2': X, 'Ordinate2': Y, 'Altitude2': Z, 'Theta': (X+Y)}
del X, Y, Z

# # Dictionnary keys are listed / the dictionnary is explicitly expressed
KeyList0 = list(MyDictionnary.keys())
print("Key list (explicitly) : {}".format(KeyList0))


## function get keys lists
def DictionnaryTest(*args):
    return(list(chain(*args)))

# A list of dictionnary names is now created
DictionnaryNamesTable = [MyDictionnary, MyDictionnary2]

KeyList2 = DictionnaryTest(*DictionnaryNamesTable)
print("Key list2 : {}".format(KeyList2))


## list composed of a single dictionnary
DictionnaryNamesTable = [MyDictionnary]
KeyList1 = DictionnaryTest(*DictionnaryNamesTable)
print("Key list1 : {}".format(KeyList1))

[ndc85430]

Dictionnaries names and numbers are variables

Again, avoid doing this because that code is more difficult to read and harder to maintain. If you have a load of things that you need to use, then put those things in a collection as buran shows.

[paul18fr]

Again, avoid doing this because that code is more difficult to read and harder to maintain. If you have a load of things that you need to use, then put those things in a collection as buran shows.

For sure your experiences and feedbacks are much greater than mine, so I'm seriously listening to your advices and warnings .

I’m using dictionaries to record and store data where dictionaries are physical quantities and Keys are components for example; values are generally updated during the recording process; as you can imagine, their names, numbers, sizes and so on vary for each study.

I used locals(), globals(), exec() so far, but as you mentioned it should be avoided, and I’m working on a (good) way to remove them: even if I’m trying to well document my codes, maintainability is ones of the weakness.

In addition, I’m working in huge amount of data (the latest file I worked on as about 1 billion lines), it’s necessary to remain “fast” and so I want to read it only ounce (currently it takes few minutes for reading and storing) .. but I'm not perfect

[buran]

I’m using dictionaries to record and store data where dictionaries are physical quantities and Keys are components for example; values are generally updated during the recording process; as you can imagine, their names, numbers, sizes and so on vary for each study.

I’m working in huge amount of data (the latest file I worked on as about 1 billion lines),

well, this screams you need database....

[paul18fr]

well, this screams you need database....

Or I can store in hdf5 files (with h5py) and then work with 2D and 3D (numpy) arrays