Trying to understand blocking in both multitasking and asyncio

[penright]

Setting the stage for the questions. I am going to explain a lot here and from now on as I ask different python questions I will just link back to here.

I am trying to write a component for Home Assistant. This is my first python project. I am very versed with .NET especially VB.NET. The module will be called form another python program. It uses websocket to talk to a Almond+ API. Right now I have it boiled down to where a CLI can exercise the functions. One other note, the Almond+ is a device that has been sunset. It main function was a WiFi router. But it also had zwave and zigbee radios. This means it could do home automation. With the Almond+ api you can command devices and receive events when states changes. A command example would to switch a state (turn on a light switch). And a event example would be when someone turn on/off a switch and the A+ api would send an event.
Back ground information that is mostly FYI but I am including it because it might be used as references.
Link to Securifi who made the Almond+ https://www.securifi.com/
Link to Almond+ api https://wiki.securifi.com/index.php/Webs...umentation
Link to Home Assistant https://www.home-assistant.io/
Link to project Github https://github.com/penright/pyalmondplus


The api is mostly be a wrapper to the Almond+. It will also track devices in the HA world is called entities.
I had a successful "hello world" api that connect and received a device list.
The api code is in pyalmondplus.py. So from now on, unless noted, when I refer to the api it is the one I am creating. Then to test the api the user can type in a CLI. That code is in cli.py.

Enough background, if you are being link here from another question, this is all of the background information.

So the bottom, coming from a different language, and from what I have read. I think my issue is (and what I am trying to do) revolves around understanding the "GIL" and (Parallel vs. Concurrent). I am/was expecting "Parallel" and the examples I been reading are "Concurrent". I played with asycio and threads. I want the almondplus.py class "start()" to "kickoff the self.receive()" and return with the api waiting for the receive.
So I expect or maybe better said, I want the output to be "Start 1", "Start 2", "receive started", "Receiver running"
If I use a self.loop.run_(forever or until) I get the "receive started" but not the "Receiver running", because the loop.run_* is blocking. Without it the receiver is never kicked off. Thanks in advance.


Here is the cli code

# -*- coding: utf-8 -*-

"""Console script for pyalmondplus."""
import sys
import time
import asyncio
import click
import pyalmondplus.api


@click.command()
@click.option('--url', default='')
def start_api(url):
    """Console script for pyalmondplus."""
    print("start_api started")
    almond_devices = pyalmondplus.api.PyAlmondPlus(url)
    almond_devices.start()
    print("Connected to Almond+")
    do_commands(url)
    return 0


def do_commands(url):
    #click.echo("Connecting to " + url)
    while True:
        time.sleep(3)
    while True:
        value = click.prompt("What next: ")
        print("command is: " + value)
        if value == "stop":
            break


def main():
    print("test")
    print("Setting up loop")
    loop = asyncio.get_event_loop()
    print("Starting loop")
    asyncio.get_event_loop().run_until_complete(start_api())
    print("loop started")
    # #main()
    # #sys.exit(main())
    print("Finishing")

And here is the api (almondplus.py)

# -*- coding: utf-8 -*-
import asyncio
import websockets
import json


class PyAlmondPlus:

    def __init__(self, api_url, event_callback=None):
        self.api_url = api_url
        self.ws = None
        self.loop = asyncio.get_event_loop()
        self.receive_task = None
        self.event_callback = event_callback

    async def connect(self):
        print("connecting")
        if self.ws is None:
            print("opening socket")
            self.ws = await websockets.connect(self.api_url)
        print(self.ws)

    async def disconnect(self):
        pass

    async def send(self, message):
        pass

    async def receive(self):
        print("receive started")
        if self.ws is None:
            await self.connect()
        recv_data = await self.ws.recv()
        print(recv_data)
        await self.receive()

    def start(self):
        print("Start 1")
        asyncio.ensure_future(self.receive(), loop=self.loop)
        print("Start 2")
        self.loop.run_forever()
        print("Receiver running")
    def stop(self):
        pass

[penright]

I did not receive and response, is there an issue with my question?

[nilamo]

I did not receive and response, is there an issue with my question?

Not all of us can get to every question right away, that doesn't mean we won't get there :p
And then also this is using asyncio, which is complicated and really hard to understand if you don't already know how it works, which limits who can help.

Did you write all that code? Is it from a guide somewhere? Does the api actually require using asyncio? I'm not sure how much benefit it actually adds to this project.

That said...

        self.loop.run_forever()
        print("Receiver running")

...Receiver running would only ever be printed once the loop has completed. ie: when it's not actually running anymore. That's because, as you noted, asyncio's event loops block the main thread once you start them, and will run whatever handlers you passed it until they complete (which might be never, for things like a server... or what you're building lol).

[penright]

Not all of us can get to every question right away, that doesn't mean we won't get there :p

Sorry, I did not mean to seem impatient. When you are new to a forum it takes a little back and forth to learn the etiquette.
That is why I was afraid I asked it the wrong way.

Did you write all that code?

I think what I here you asking is do I understand what I type or did I just copy it and went hum....
Can I say a little of both. Mostly I created it.

Does the api actually require using asyncio?

I going to say a definitely maybe. Ok, that was attempted humor. The reason it was attempted, because it did not quit make it.
It started with, I needed to connect to a websocket. The example used Websocket 5.0 and it was setup as an asycio. That was my introduction to asyncio. That all worked and I thought I understood how asyncio worked. So I wanted a way to instance the class and get user input. That when I learned about Click and entry points to create a CLI (command line interface). So coming from the VB.NET world, I thought each loop was a thread. I have read enough of asyncio now to know it runs in the same thread. FYI, what will be instancing the class will pass it's asyncio loop. Then I learn that 3.5 up, you don't have to. They require >3.5, but still pass the loop for backward compatibility. That why I was trying to emulate it in my CLI.

I'm not sure how much benefit it actually adds to this projec

Again, the CLI is just to test the API. The end result is to create what is called a "component" for the Home Assistant. A component is kind of a plugin for different devices. Here is an example of some that other has done so far .... https://www.home-assistant.io/components/ New ones are getting added every release, which is about two week cycle. It also supports a "custom" directory that you copy a non merged component into it and now it is part of the code.
I have not dug into how all that glue works. I do know I need to handle asynchronous messages from the Almond+ and I assume from the Home Assistant (HA). I was thinking if I had a loop running the CLI, that could tickle and consume events for the API, then I could translate that knowledge into what the component would need.

Again, bottom line, I trying to have two loops. One running in the CLI that taking user input and calling a function in the API. The second is the receive loop running the API for events from the Almond+

Once I get to that point, I will make the receive more intelligent. The when sending a command to the A+ (Almond+), there is a ID that is part of the message and the A+ will echo it back as part of the response. So the receive is kind of a dispatcher based on the message. It can match it up to a sent message for a return value for what send function. And it could generate an event for events.

Again, first step is bottom line. Set up a loop taking input that the receive in the api can "cooperate" with. Instead of

[penright]

Made some progress. I may have had it the first time I tried.
Lessons learned ....
1. I forgot I was using entry point main(), and I thought the proper way to start was the pattern "if __name__ == main()" So when I tried to start the loop in that section, I was not getting what I expected. duh ...
2. I thought any io would be async with asycnio. So the click.prompt was blocking. But when I tested with time.sleep() it blocks also.

I still have not met the goal of this step yet. I cheated the input for asycnio sleep/print. This was to prove my logic and to trouble shoot the code down to the input being the blocking. Now I need a asyncio std in/out. But's another thread.

Here is the code and output of where I am at right now.

The CLI

# -*- coding: utf-8 -*-

"""Console script for pyalmondplus."""
import sys
import time
import asyncio
import click
import pyalmondplus.api



def start_api(url):
    """Console script for pyalmondplus."""
    print("getting loop")
    loop = asyncio.get_event_loop()
    print("start_api started")
    asyncio.ensure_future(do_commands(url))
    print("Loop Started")
    loop.run_forever()
    loop.close()


async def do_commands(url):
    print("Do commands 1")
    almond_devices = pyalmondplus.api.PyAlmondPlus(url)
    print("Do commands 2")
    almond_devices.start()
    print("Connected to Almond+")
    while True:
        await asyncio.sleep(3)
        print("do command is running")
    # click.echo("Connecting to " + url)
    # while True:
    #     value = await click.prompt("What next: ")
    #     print("command is: " + value)
    #     if value == "stop":
    #         break


@click.command()
@click.option('--url', default='')
def main(url):
    start_api(url)

The API

# -*- coding: utf-8 -*-
import asyncio
import websockets
import json


class PyAlmondPlus:

    def __init__(self, api_url, event_callback=None):
        self.api_url = api_url
        self.ws = None
        self.loop = asyncio.get_event_loop()
        self.receive_task = None
        self.event_callback = event_callback

    async def connect(self):
        print("connecting")
        if self.ws is None:
            print("opening socket "+self.api_url)
            self.ws = await websockets.connect(self.api_url)
        print(self.ws)

    async def disconnect(self):
        pass

    async def send(self, message):
        pass

    async def receive(self):
        print("receive started")
        if self.ws is None:
            await self.connect()
        while True:
            recv_data = await self.ws.recv()
            print(recv_data)

    def start(self):
        print("Start 1")
        asyncio.ensure_future(self.receive(), loop=self.loop)
        print("Start 2")
        print("Receiver running")

    def stop(self):
        pass

The output:

(pyalmondplus) Pauls-MBP:pyalmondplus paulenright$ testapi --url ws://192.168.1.2:7681/root/xxxxxx
getting loop
start_api started
Loop Started
Do commands 1
Do commands 2
Start 1
Start 2
Receiver running
Connected to Almond+
receive started
connecting
opening socket ws://192.168.1.2:7681/root/xxxxxx
<websockets.client.WebSocketClientProtocol object at 0x101e38208>
{"CommandType":"DynamicAlmondModeUpdated","Mode":"2","EmailId":"[email protected]"}
do command is running
do command is running
do command is running
{"CommandType":"DynamicIndexUpdated","Devices":{"2":{"DeviceValues":{"1":{"Name":"SWITCH_BINARY1","Value":"true"}}}}}
{"CommandType":"DynamicSceneActivated","Action":"update","Scenes":{"3":{"Active":"true","LastActiveEpoch":"1530104094"}}}
{"CommandType":"DynamicSceneActivated","Action":"update","Scenes":{"4":{"Active":"false","LastActiveEpoch":"1530105795"}}}
do command is running
do command is running
{"CommandType":"DynamicIndexUpdated","Devices":{"2":{"DeviceValues":{"1":{"Name":"SWITCH_BINARY1","Value":"false"}}}}}
do command is running
{"CommandType":"DynamicSceneActivated","Action":"update","Scenes":{"3":{"Active":"false","LastActiveEpoch":"1530105802"}}}
{"CommandType":"DynamicSceneActivated","Action":"update","Scenes":{"4":{"Active":"true","LastActiveEpoch":"1530105795"}}}
do command is running
do command is running
^C
Aborted!
(pyalmondplus) Pauls-MBP:pyalmondplus paulenright$

[nilamo]

Oh, YOU'RE penright lol. I looked up the github repo to see what the api was doing, and was going to ask why your class had the same names as the one in the repo... and then I realized it's because it's all you lol.

I'm trying to get caught up, but there seems like a whole lot of moving parts to this thing.

I'm still digging, while doing other things, but for others, here's the components that ship with home-assistant (hopefully the github repo can let us dig into how to do something similar ourselves): https://github.com/home-assistant/home-a...components

[penright]

I'm trying to get caught up, but there seems like a whole lot of moving parts to this thing

I have learned a lot since my first post. I starting to see the issues.
Looks like websockets need asyncio. Here is my current CLI, API, and Output. https://hastebin.com/cazizapafe.py
This was attempt to run without asyncio. The CLI started two threads, one for commands and the other for the API.
The API threw an error when the websocket tried to connect saying there was not event_loop.
My next try will be 100% asyncio. I need to be able to emulate the command loop without blocking the event_loop.

Another tack I have not gave up on totally, is launching the event_loop in a different thread. For some reason even if in a different thread, as soon as I run the event_loop it blocks other threads.
I think I have it boiled down to an example code. All this to help me understand where threads are getting blocked.
As posted the event_loop is rem out in the loop_helper function.
There are two way of lunching the main(). 1. is by an entry point created with "setup.py devlop", the second is running it from the interpreter. Right now I an testing in PyCharm.
So the work is in main(). There are two threads started from main(). There is a class "Test" that starts a event_loop "start()->Thread(target=loop_helper)->loop_work. loop_work shows the third thread is running.
When the event_loop is no rem out, I expect the loop_helper() to print "test from helper" and return, then the start() print "Receiver running" and return. Then the main() will fall into it's loop. It does not, the loop.run_until blocks.

Here is the boiled down code and output. If I can get the event_loop to run and not block the other threads, I can make the API work.
https://hastebin.com/qayataqoxi.sql

[penright]

SOLVED

Got it.
https://hastebin.com/texegomedo.py
This was the test that emulated the CLI and the API. Which is emulated by the Test class. The Test .start() and .stop() are ran and the .start returns back so the loop in the main() is running.
The issue was how I was starting the thread. When I specified the target= I had () on the end of the function. That explains why the "function" was running even before the theard.start.
Python was evaluating the function and not being passed as a object.
I am so glad to discover code still is not magic. Man that makes so much sense now.

Now I hope I can translate it to the CLI and API.