TypeError: missing 3 required positional arguments:

[wardancer84]

hi all,

I got the error "TypeError: missing 3 required positional arguments:". I am aware what this usually means, but in that case...not.

Here is the code, just a simple udp server.

class AIX_REGISTRY_UDPRequestHandler(socketserver.DatagramRequestHandler):
    def handle(self):
        r = self.open_db_connection(**db_config)
        c = r[0]
        cur = r[1]
        for line in self.rfile:
            data = line.strip()
            data = data.decode('utf-8')
        cur_thread = threading.current_thread()
        logger.debug("Thread: {} - Message:{}".format(cur_thread.name, data))

        version, node_name, attrib_name, value = data.split("|")
        nodes = self.reload_nodes(cur)
        attribs = self.reload_attribs(cur)

        sql = "INSERT INTO entries VALUES (DEFAULT, %d, %d, '%s', DEFAULT)" % (
                                                                               self.get_or_create_node_id(node_name,c,cur,nodes),
                                                                               self.get_or_create_attrib_id(attrib_name,c,cur,attribs),
                                                                               value,
                                                                              )
        try:
            cur.execute(sql)
            c.commit()
        except db.MySQLError as e:
            logger.error("Something went wrong during db interaction: {}".format(e))
        finally:
            if cur is not None:
                cur.close()
            if c is not None:
                c.close()

    def open_db_connection(self,**arg):
        """Connect to MySQL Database"""
        try:
            c = db.connect(**arg)
            cur = c.cursor()
            #logger.info("Connection to db opened successfully.")
            return [c, cur]
        except db.MySQLError as e:
            logger.error("Error while trying to connect to db {}".format(e))


    def reload_nodes(self,cur):
        sql = "SELECT name,id from nodes"
        row = cur.execute(sql)
        result = cur.fetchall()
        return dict(result)

    def reload_attribs(self,cur):
        sql = "SELECT name,id from attribs"
        row = cur.execute(sql)
        result = cur.fetchall()
        return dict(result)

    def get_or_create_node_id(self,node_name,c,cur,nodes):
        if node_name in nodes:
            print ("%s found in dict, has id %s" % ( node_name, nodes[node_name] ))
            return int(nodes[node_name])
        else:
            print("creating new node entry ..")
            sql = "INSERT INTO nodes VALUES (DEFAULT, '%s')" % (node_name)
            cur.execute(sql)
            c.commit()
            return self.get_or_create_node_id(node_name)

    def get_or_create_attrib_id(self,attrib_name,c,cur,attribs):
        if attrib_name in attribs:
            print ("%s found in dict, has id %s" % ( attrib_name, attribs[attrib_name] ))
            return int(attribs[attrib_name])
        else:
            print("creating new attrib entry ..")
            sql = "INSERT INTO attribs VALUES (DEFAULT, '%s', DEFAULT, DEFAULT)" % (attrib_name)
            cur.execute(sql)
            c.commit()
            return self.get_or_create_attrib_id(attrib_name)

traceback:

creating new node entry ..
----------------------------------------
Exception happened during processing of request from ('172.17.10.20', 41963)
Traceback (most recent call last):
  File "/opt/freeware/lib64/python3.7/socketserver.py", line 650, in process_request_thread
    self.finish_request(request, client_address)
  File "/opt/freeware/lib64/python3.7/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/opt/freeware/lib64/python3.7/socketserver.py", line 720, in __init__
    self.handle()
  File "./aix_reg_server.py_needs_rework", line 61, in handle
    self.get_or_create_node_id(node_name,c,cur,nodes),
  File "./aix_reg_server.py_needs_rework", line 108, in get_or_create_node_id
    return self.get_or_create_node_id(node_name)
TypeError: get_or_create_node_id() missing 3 required positional arguments: 'c', 'cur', and 'nodes'

the funny thing is the are arguments are clearly passed to function self.get_or_create_node_id. beside that, it seems it iterates over the arguments list somehow, as 4 entries instead of 1 are created in the database.

MariaDB [aix_registry]> select * from nodes where name like '%AIXTESTLPAR%';
+------+---------------+
| id   | name          |
+------+---------------+
| 2219 | AIXTESTLPAR01 |
| 2220 | AIXTESTLPAR01 |
| 2221 | AIXTESTLPAR01 |
| 2222 | AIXTESTLPAR01 |
+------+---------------+
4 rows in set (0.001 sec)

any ideas?

wbr

chris

[wardancer84]

got it, line 64 and 75 are missing the args

[deanhystad]

This code makes no sense to me. What do you expect it to do?

    def get_or_create_node_id(self,node_name,c,cur,nodes):
        if node_name in nodes:
            print ("%s found in dict, has id %s" % ( node_name, nodes[node_name] ))
            return int(nodes[node_name])
        else:
            print("creating new node entry ..")
            sql = "INSERT INTO nodes VALUES (DEFAULT, '%s')" % (node_name)
            cur.execute(sql)
            c.commit()
            return self.get_or_create_node_id(node_name)

[wardancer84]

This code makes no sense to me. What do you expect it to do?

    def get_or_create_node_id(self,node_name,c,cur,nodes):
        if node_name in nodes:
            print ("%s found in dict, has id %s" % ( node_name, nodes[node_name] ))
            return int(nodes[node_name])
        else:
            print("creating new node entry ..")
            sql = "INSERT INTO nodes VALUES (DEFAULT, '%s')" % (node_name)
            cur.execute(sql)
            c.commit()
            return self.get_or_create_node_id(node_name)

it looks in the database of the "node" is already there, if yes it returns the id of the node (db id), if no it creates the node in the db.

[deanhystad]

Why the recursion? I only see that being a problem because you'll somehow have to get an updated "nodes" or it will keep calling itself until it reaches the recursion limit and crashes.

[wardancer84]

Why the recursion? I only see that being a problem because you'll somehow have to get an updated "nodes" or it will keep calling itself until it reaches the recursion limit and crashes.

the recursion part was added by the original author and he told me this is absolutely necessary. but it seems not working now...i get strange results.

multiple inserts in the database, like some loop is going wild...seems your are asking the right questions

creating new node entry ..
creating new node entry ..
creating new node entry ..
creating new node entry ..
creating new node entry ..
creating new node entry ..
creating new node entry ..
AIXTESTLPAR01 found in dict, has id 737
Java8_64 found in dict, has id 61
AIXTESTLPAR01 found in dict, has id 737
HAS_SAP found in dict, has id 36
creating new node entry ..
AIXTESTLPAR01 found in dict, has id 737
IP_LONG found in dict, has id 85
AIXTESTLPAR01 found in dict, has id 738
HAS_UC4 found in dict, has id 95
creating new node entry ..
AIXTESTLPAR01 found in dict, has id 740
IP found in dict, has id 84
AIXTESTLPAR01 found in dict, has id 737
AIXTESTLPAR01 found in dict, has id 738
CPU_VALUE found in dict, has id 37
IS_CLUSTER found in dict, has id 26
creating new node entry ..
AIXTESTLPAR01 found in dict, has id 743
AIXTESTLPAR01 found in dict, has id 739
AIXTESTLPAR01 found in dict, has id 738
AIXTESTLPAR01 found in dict, has id 737

[deanhystad]

I think you must have copied the code wrong. As is, the recursion makes no sense at all.

The recursion makes sense, and missing arguments too, if the method was written to query the database to get an updated nodes dictionary, test if the new node is in the dictionary, add the node if it isn't, then recursively call itself to query the database and update the dictionary. Something like this:

def get_or_create_node_id(self, node_name):
    if node_name in self.reload_nodes():  # This updates the nodes dictionary
        return int(nodes[node_name])
    else:
        sql = "INSERT INTO nodes VALUES (DEFAULT, '%s')" % (node_name)
        cur.execute(sql)
        c.commit()
        return self.get_or_create_node_id(node_name)

A problem with this is that you make a lot of calls to self.reload_nodes(). It might make more sense to organize the code like this:

def get_or_create_node_id(self, node_name):
    if node_name not in self.nodes:
        sql = "INSERT INTO nodes VALUES (DEFAULT, '%s')" % (node_name)
        self.cur.execute(sql)  # Should be an instance variable
        self.c.commit()  # Should be an instance variable
        self.update_nodes()
    return int(self.nodes[node_name])

And modify update_nodes() to update an instance variable.

    def reload_nodes(self):
        sql = "SELECT name,id from nodes"
        row = sself.cur.execute(sql)   # cur should also be an instance variable instead of an argument
        self.nodes = dict(self.cur.fetchall())
        return self.nodes

[wardancer84]

good points, heres the full code with all your proposals implemented.

#!/usr/bin/env python3

import threading
import socketserver

import os
import sys
import errno
import pymysql as db
import logging

#
# logger & PID setup
#

pid_file = "/tmp/registry_server_dev.pid"
log_file = "/tmp/aix_registry_server_dev.log"

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.propagate = False
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
fh = logging.FileHandler(log_file, "w")
fh.setLevel(logging.DEBUG)
fh.setFormatter(formatter)
logger.addHandler(fh)
keep_fds = [fh.stream.fileno()]

#
# various config itmes
#

db_config = {
        'user':       'aix_reg_dev',
        'password':   'aix_reg_dev',
        'db':         'aix_registry_dev',
        'host':       'localhost',
        'port':        3306
    }


UDPServerAddress = ("nimvie_a", 6790)


class AIX_REGISTRY_UDPRequestHandler(socketserver.DatagramRequestHandler):
    def handle(self):
        self.r = self.open_db_connection(**db_config)
        self.c = self.r[0]
        self.cur = self.r[1]
        self.data = self.rfile.readline().strip()
        self.data = self.data.decode('utf-8')
        self.cur_thread = threading.current_thread()
        logger.debug("Thread: {} - Message:{}".format(self.cur_thread.name, self.data))

        version, node_name, attrib_name, value = self.data.split("|")
        self.nodes = self.reload_nodes()
        self.attribs = self.reload_attribs()

        sql = "INSERT INTO entries VALUES (DEFAULT, %d, %d, '%s', DEFAULT)" % (
                                                                               self.get_or_create_node_id(node_name),
                                                                               self.get_or_create_attrib_id(attrib_name),
                                                                               value,
                                                                              )
        try:
            self.cur.execute(sql)
            self.c.commit()
        except db.MySQLError as e:
            logger.error("Something went wrong during db interaction: {}".format(e))
        finally:
            if self.cur is not None:
                self.cur.close()
            if self.c is not None:
                self.c.close()

    def open_db_connection(self,**arg):
        """Connect to MySQL Database"""
        try:
            c = db.connect(**arg)
            cur = c.cursor()
            #logger.info("Connection to db opened successfully.")
            return [c, cur]
        except db.MySQLError as e:
            logger.error("Error while trying to connect to db {}".format(e))


    def reload_nodes(self):
        sql = "SELECT name,id from nodes"
        row = self.cur.execute(sql)   # cur should also be an instance variable instead of an argument
        self.nodes = dict(self.cur.fetchall())
        return self.nodes

    def reload_attribs(self):
        sql = "SELECT name,id from attribs"
        row = self.cur.execute(sql)
        self.attribs = dict(self.cur.fetchall())
        return self.attribs

    def get_or_create_node_id(self,node_name):
        if node_name not in self.nodes:
            print("creating new node entry {}".format(node_name))
            sql = "INSERT INTO nodes VALUES (DEFAULT, '%s')" % (node_name)
            self.cur.execute(sql)  # Should be an instance variable
            self.c.commit()  # Should be an instance variable
            self.reload_nodes()
        return int(self.nodes[node_name])

    def get_or_create_attrib_id(self,attrib_name):
        if attrib_name not in self.attribs:
            print("creating new attribs entry {}".format(attrib_name))
            sql = "INSERT INTO attribs VALUES (DEFAULT, '%s')" % (attrib_name)
            self.cur.execute(sql)  # Should be an instance variable
            self.c.commit()  # Should be an instance variable
            self.reload_attribs()
        return int(self.attribs[attrib_name])


def main():

    logger.info("Started aix_registry_server_dev with PID {}".format(os.getpid()))
    logger.info("PID file is: {} ".format(pid_file))
    logger.info("LOG file is: {} ".format(log_file))


    try:
        AIX_REGISTRY_UDPServer = socketserver.ThreadingUDPServer(UDPServerAddress, AIX_REGISTRY_UDPRequestHandler)
        AIX_REGISTRY_UDP_Thread = threading.Thread(target=AIX_REGISTRY_UDPServer.serve_forever)
        #AIX_REGISTRY_UDP_Thread.daemon = True
        AIX_REGISTRY_UDP_Thread.start()
        logger.info("UDP Listener on:{}':'{}".format(UDPServerAddress[0], UDPServerAddress[1]))
        logger.info("Server loop running in thread:{}".format(AIX_REGISTRY_UDP_Thread.name))
    except Exception as e:
        logger.error("Error while starting up: {}".format(e))
        logger.error("EXIT")
        sys.exit(1)

if __name__ == '__main__':

    main()
    #daemon = Daemonize(app="aix_registry_server", pid=pid_file, action=main, foreground=False, keep_fds=keep_fds, auto_close_fds=False)
    #daemon.start()

alas the multiplied entries remains...seems some kind of logic error in context to threading. with every run it stacks up even more. strange. i even do a manual delete in the db between the runs.

first run

MariaDB [aix_registry_dev]> select * from nodes where name like '%AIXTESTLPAR%';
+-----+---------------+
| id  | name          |
+-----+---------------+
| 824 | AIXTESTLPAR01 |
| 825 | AIXTESTLPAR01 |
| 826 | AIXTESTLPAR01 |
| 827 | AIXTESTLPAR01 |
+-----+---------------+
4 rows in set (0.001 sec)

second run

MariaDB [aix_registry_dev]> select * from nodes where name like '%AIXTESTLPAR%';
+-----+---------------+
| id  | name          |
+-----+---------------+
| 828 | AIXTESTLPAR01 |
| 829 | AIXTESTLPAR01 |
| 830 | AIXTESTLPAR01 |
| 831 | AIXTESTLPAR01 |
| 832 | AIXTESTLPAR01 |
| 833 | AIXTESTLPAR01 |
| 834 | AIXTESTLPAR01 |
| 835 | AIXTESTLPAR01 |
| 836 | AIXTESTLPAR01 |
| 837 | AIXTESTLPAR01 |
+-----+---------------+
10 rows in set (0.002 sec)

third run

MariaDB [aix_registry_dev]> select * from nodes where name like '%AIXTESTLPAR%';
+-----+---------------+
| id  | name          |
+-----+---------------+
| 838 | AIXTESTLPAR01 |
| 839 | AIXTESTLPAR01 |
| 840 | AIXTESTLPAR01 |
| 841 | AIXTESTLPAR01 |
| 842 | AIXTESTLPAR01 |
| 843 | AIXTESTLPAR01 |
| 844 | AIXTESTLPAR01 |
| 845 | AIXTESTLPAR01 |
| 846 | AIXTESTLPAR01 |
| 847 | AIXTESTLPAR01 |
| 848 | AIXTESTLPAR01 |
| 849 | AIXTESTLPAR01 |
| 850 | AIXTESTLPAR01 |
| 851 | AIXTESTLPAR01 |
| 852 | AIXTESTLPAR01 |
| 853 | AIXTESTLPAR01 |
| 854 | AIXTESTLPAR01 |
| 855 | AIXTESTLPAR01 |
| 856 | AIXTESTLPAR01 |
| 857 | AIXTESTLPAR01 |
| 858 | AIXTESTLPAR01 |
| 859 | AIXTESTLPAR01 |
| 860 | AIXTESTLPAR01 |
| 861 | AIXTESTLPAR01 |
| 862 | AIXTESTLPAR01 |
+-----+---------------+
25 rows in set (0.001 sec)

[wardancer84]

after removing threading stuff everything is working normal...i guess i will drop threads, as pymysql is not threadsafe anyway.

[deanhystad]

Looks like keeping a local copy of the nodes dictionary is a bad idea.