Queuing the file content and run the thread.

[pythonlover]

Hello,
I have a list of network devices in a file (each line is a device). I am passing each device or each line of that file as an argument to a curl statement to get the interfaces from the devices. The problem is, since there are more than 20K devices - it is actually taking a lot of time to get all the interfaces. 
This led me into threading and queuing module. Now, I am queuing all the devices and then running the thread. The thread contains the function to pull the interfaces. 

with open("filtered_devices") as my_file:
   for line in my_file:
     q.put(line)

numofThreads = 10.
for i in range (numofThreads):
   print str(threading.activeCount())
   t = threading.Thread(target = queue_interfaces())
   t.start()
   #threadlist.append(t)

Here, queue_interfaces() is the function that passes each devices as as argument and gets the interfaces. Not sure if this is the correct way to do that. May be I am doing it wrong, since the time is still the same. Can somebody please advise?
Also, my active thread count is not incrementing.

[micseydel]

Are you joining on the queue?

[pythonlover]

Yes, I am.

q.join()

[micseydel]

Can you provide enough code to reproduce the problem? You can reply the devices by curling Google or something. Also, I just noticed that the code you've posted passes a float to range() which is not valid. Don't take this personally, but oftentimes people post code that isn't terribly related to their problem, and it really helps us to get real runnable code. Otherwise it can be very difficult to help.

[pythonlover]

I am sorry about that. Not sure how 10 became a floating - it was "10" when I posted. Here's my complete code. I have not included the url but you can assume it is an API call. Also, I have a list of devices in the file "filtered_devices".

import urllib
import urllib2
import requests
import threading
import Queue

#Queuing
q = Queue.LifoQueue()

url = 'x'


def queue_interfaces(q):    #grabs the queue and starts getting the interfaces from the list
    while not q.empty():
        headers = {"Content-Type": "application/json"}
        a = str(q.get()).strip('\n')
        data = {"data":{"Type":"Interfaces","Filters":{"device__name":a},"Attributes":["speed","status__name"],"page_size":10}}
        r = requests.post(url, headers=headers, json=data)
        #print (r.text)
        f = open("interfaceslistfinal", 'w')
        f.write(r.text)
        q.task_done()
    f.close()


with open("filtered_devices") as my_file:
    for line in my_file:
      q.put(line)

q.join()

#Threading
numofThreads = 10
#threadlist = []
for i in range (numofThreads):
    print i
    t = threading.Thread(target = queue_interfaces, args=(q,))
    t.setDaemon(True)
    t.start()

[micseydel]

I'm no expert on threading, but I'm fairly certain that at least one problem is that you need to join on the queue after starting the threads.

Another, more minor issue that probably isn't causing issues, is that f.close() should be inside your while-loop.

I don't have a chance at the moment, but I'll play with this code when I get a chance.

[pythonlover]

Cool, Thanks micseydel. Despite joining the queue after starting the threads, it still doesn't do that good. The time is still the same. Yeah, f.close () is inside the while-loop. Please let me know when you get a chance to play around and find something that can help.

[micseydel]

Apologies for taking so long to get back, but I've been able to reproduce the issue very concisely

import Queue
q = Queue.LifoQueue()
q.put(None)
q.join()

This will hang. Here is the same thing with more prints

import Queue
q = Queue.LifoQueue()
print "Putting into the queue"
q.put(None)
print "About to join()"
q.join()
print "Done with join()"

Basically, the problem is q.join() will block until all the items in the queue have been completed. More specifically, the docs say that task_done() being called is required to resolve items, but when you join on the queue you haven't started any threads yet.

I haven't dug deeper into your code, but I'll let you integrate this for now. If you have followup questions I'm still happy to come back to this, though as you know that could take some time....

[pythonlover]

Thanks a lot [b]micseydel ! [/b]BTW, I have already fixed my code. Sorry, that I didn't update the thread. [Part of my code:]

# Queuing
q = Queue.Queue(maxsize=0)
lock = threading.Lock()   # Lock the file while writing for 1 thread

#Threading
with open("filtered_devices") as my_file:
    for line in my_file:
      q.put(line)

numofThreads = 20

for i in range (numofThreads):
    print "Threading Count No:%d" %i
    t = threading.Thread(target = queue_interfaces, args=(q,))
    t.setDaemon(True)
    t.start()

q.join()