Open Broken connection to Adafruit IO?

Discussion in 'Join' started by pedalermike, Jul 17, 2018.

  1. pedalermike

    pedalermike New Member

    Joined:
    Jul 17, 2018
    Messages:
    9
    Likes Received:
    0
    I've been a paid user of Tasker/ Join for several years and developed a comprehensive Arduino IoT smart house solution using Adafruit's IO MQTT broker. A part of the client side app periodically send status updates to the broker which then reformats it according to the join api and sends it to my phone via a join push. (My Arduino has access to the internet but the processor doesn't have enough resources to connect via https, so I had to split the status code into a client side/server side and let the server side handle the https push). That was working for over a year.

    Recently (last Friday) the connection between Adafruit IO and Join API stopped working. I've checked with the Adafruit IO tech team and no system changes were made on their side. In diagnosing the problem, I downloaded the join push requests from Adafruit MQTT broker and manually tested them in my PC's Chrome browser. All the pushes worked as they should.

    I then modified the Adafruit broker-side code to send the status updates to my IFTTT maker channel (instead of the join API) and from IFTTT, post the join push using webhooks. That works.

    When I take the same join push and have it posted directly by the Adafruit broker, it doesn't go through. Since the only difference I can see is that the originating IP addresses (Adafruit broker vs IFTTT webhooks) are different, it is possible that the server at the join api is blocking connections from the Adafruit broker?

    I have very little in the way of audit trails or debugging tools at either Adafruit or IFTTT so everything is being figured out as though Join, IFTTT, Adafruit are black boxes.
     
  2. joaomgcd

    joaomgcd Administrator Staff Member

    Joined:
    Feb 3, 2015
    Messages:
    9,477
    Likes Received:
    806
    Hi there. Are there any errors log that you can check in the Adafruit system so you can figure out what the error is? Thanks
     
  3. pedalermike

    pedalermike New Member

    Joined:
    Jul 17, 2018
    Messages:
    9
    Likes Received:
    0
    The Adafruit side of this post is located here: https://forums.adafruit.com/viewtopic.php?f=56&t=138202&p=684151#p684151. It doesn't appear that there are any logging capabilities on their end. IFTTT does log but the info is very high level and other than confirming that the join push works as it is supposed to, there isn't much more that it can offer. It does seem to be IP address related though...
     
  4. joaomgcd

    joaomgcd Administrator Staff Member

    Joined:
    Feb 3, 2015
    Messages:
    9,477
    Likes Received:
    806
    Do you know how exactly that service makes their requests? It's going to be really hard to figure out the issue if they don't offer some kind of error logging :(
     
  5. pedalermike

    pedalermike New Member

    Joined:
    Jul 17, 2018
    Messages:
    9
    Likes Received:
    0
    I agree. The lack of a logging feature on their side is very frustrating. I created a new account on Adafruit IO last night and configured it with the same join API push that used to work and when I activated it, the transaction never showed up (or perhaps was ignored) by join. I think this eliminates the possibility of an account configuration error.... I'm going to ask them to ping joinjoaomgcd.appspot.com from their server and see if they get a response. Can you confirm that the join API IP address is 172.217.0.244?
     
  6. joaomgcd

    joaomgcd Administrator Staff Member

    Joined:
    Feb 3, 2015
    Messages:
    9,477
    Likes Received:
    806
  7. pedalermike

    pedalermike New Member

    Joined:
    Jul 17, 2018
    Messages:
    9
    Likes Received:
    0
    Weird, I'm getting a different address for joinjoaomgcd.appspot.com --> 108.177.122.153
     
  8. pedalermike

    pedalermike New Member

    Joined:
    Jul 17, 2018
    Messages:
    9
    Likes Received:
    0
    Well, had some success in getting Adafruit IO to open up the logging file. A screen shot is attached. When I copy and paste the string into a browser, it works fine. When it is initiated from the Adafruit broker, it gets a 400 error as shown.... Suggestions what look for next? Trigger Log.png
     
    Last edited: Jul 20, 2018
  9. joaomgcd

    joaomgcd Administrator Staff Member

    Joined:
    Feb 3, 2015
    Messages:
    9,477
    Likes Received:
    806
    It seems that the server is trying to send a POST request with some data where it should simply be doing a GET request. Do you know why that is?
     
  10. pedalermike

    pedalermike New Member

    Joined:
    Jul 17, 2018
    Messages:
    9
    Likes Received:
    0
    I will find out..
     
  11. pedalermike

    pedalermike New Member

    Joined:
    Jul 17, 2018
    Messages:
    9
    Likes Received:
    0
    Here is the reply from Adafruit... in short, it looks like nothing changed on their end so I'm still mystified why my webhooks are getting rejected where they were working before... I do like the thought though of a direct link between Join and Adafruit IO... there is already one between IFTTT and Adafruit ....

    "Using HTTP POST when sending webhook data to an external receiving service is (as far as I know) a standard practice across the industry. For example, SendGrid https://developer.github.com/webhooks/ and GitHub https://sendgrid.com/blog/whats-webhook/ are two pretty significant developer-focused services that do the same. While our service is nowhere near as well documented as theirs, we also use POST requests since our webhooks include data in the body of the request. This doesn't mean it's a universal practice, though. We are unlikely to add an option to change the request type in the near future, but we can provide better documentation.

    Also, I'm not familiar with Join Server / joinjoaomgc.appspot.com, it's possible if it's an open source project we could add a hook for Adafruit IO.

    Two alternatives:

    If you are able to use a service like Glitch, I built a tiny webhook receiver app to test our service. That glitch app can be remixed to proxy requests and translate HTTP POST + JSON request body to HTTP GET w/ URL params. If you're interested, I can add some code to that app to show what proxying requests through Glitch would look like. Honestly, it'd make a pretty good Learn guide, so I'll probably get to it eventually anyways.

    I also am a big fan of https://ngrok.com/ which is kind of a global-to-local proxy / tunnel that can be used to connect an external URL (provided by ngrok, where you would receive a webhook request) to a service running on your local machine. I use it for Adafruit IO development to test external services that use webhooks to communicate--like Stripe, our payment provider--against the development version of our code.

    Both of these are free services but require a bit of web backend programming. "
     
  12. joaomgcd

    joaomgcd Administrator Staff Member

    Joined:
    Feb 3, 2015
    Messages:
    9,477
    Likes Received:
    806
    Ok, I think I know what the issue is.

    I recently did a mandatory update to the server where I had to upgrade the version of the backend software it's running on. The issue is that Adafruit is sending an array in the request body and apparently the new version of the backend doesn't like that...

    If only the request didn't have that [] around the JSON stuff would work :)

    Do you think there's any chance they can change that? Or does it make sense for them to make the body an array for some reason?
     
  13. pedalermike

    pedalermike New Member

    Joined:
    Jul 17, 2018
    Messages:
    9
    Likes Received:
    0
    It seems that modifying their webhook call would make it inconsistent with the industry accepted practice. If I were them, I wouldn't do it. It seems I have a couple of routes to go:

    1. Continue routing through IFTTT with the inherent lag and complexity
    2. Use one of the proxy services that Adafruit suggested

    Alternatively, a few more questions:

    a. Why your update was not regression tested against the industry POST practice? Is there a security issue?
    b. Would you be interested in providing a direct api to Adafruit IO similar to what you do for IFTTT?

    tnx
    Mike
     
  14. joaomgcd

    joaomgcd Administrator Staff Member

    Joined:
    Feb 3, 2015
    Messages:
    9,477
    Likes Received:
    806
    Can you clarify what you mean by "industry accepted practice"? Why would sending a JSON object instead of an array not be an industry accepted practice exactly?

    About your questions:
    a. Because I didn't know people were sending JSON arrays to the endpoints. Apparently google cloud endpoints don't think it's standard unfortunately.
    b. Unfortunately that's too niche a product for me to invest time in I believe. Sorry :(

    Couldn't they make it optional to send a JSON object instead of an array? Why are they sending an array if it contains only one item? Thanks again
     

Share This Page