AutoTools Help parsing xml using AutoTools

Discussion in 'AutoApps' started by jrobbom5, Dec 12, 2018.

  1. jrobbom5

    jrobbom5 New Member

    Joined:
    Dec 12, 2018
    Messages:
    8
    Likes Received:
    0
    Hi all,

    First-time poster, please be gentle.....

    I am trying to parse this short XML download from Australias Bureau of Meteorology website, do a bit of cutting and dicing, and then export what's left as JSON:

    ftp://ftp.bom.gov.au/anon/gen/fwo/IDV10450.xml

    There are 2 sections product.forecast.area.forecast-period which contain index 0-7. I am trying to combine those 2 sections, and discard everything else. Here is what I have:


    Test BOM (35)
    A1: Run Shell [ Command:ftpget ftp.bom.gov.au /sdcard/Vibrono/Weather.xml /anon/gen/fwo/IDV10450.xml Timeout (Seconds):0 Use Root:Off Store Output In: Store Errors In: Store Result In: ]
    A2: AutoTools Json Read [ Configuration:Input Format: XML
    Json: /sdcard/Vibrono/Weather.xml
    Json Root Variable: %weather
    Separator: , Timeout (Seconds):60 ]
    A3: AutoTools Json Read [ Configuration:Json: %weather
    Fields: product.forecast.area()
    Variable Name: fred()
    Separator: , Timeout (Seconds):60 ]
    A4: For [ Variable:%num Items:1:%fred(#) ]
    A5: Array Clear [ Variable Array:%forecastperiod ]
    A6: AutoTools Json Read [ Configuration:Input Format: Json
    Json: %fred(%num)
    Fields: aac,forecast-period()
    Separator: , Timeout (Seconds):60 ]
    A7: If [ %forecastperiod(#) > 1 ]
    A8: For [ Variable:%fpnum Items:1:%forecastperiod(#) ]
    A9: AutoTools Json Read [ Configuration:Input Format: Json
    Json: %forecastperiod(%fpnum)
    Fields: index
    Separator: , Timeout (Seconds):60 ]
    A10: Variable Set [ Name:%arraykey To:fperiod%index Recurse Variables:Off Do Maths:Off Append:Off ]
    A11: AutoTools Json Write [ Configuration:Separator: ,
    Json Input: %myjsonresult
    Prettify: true
    Add to Array Key: forecast.%arraykey
    Array Object Values: %forecastperiod(%fpnum)
    Arrays Separator: | Timeout (Seconds):60 ]
    A12: Variable Set [ Name:%myjsonresult To:%atjsonresult Recurse Variables:Off Do Maths:Off Append:Off ]
    A13: End For
    A14: End If
    A15: End For
    A16: [X] Flash [ Text:%atjsonresult Long:Off ]
    A17: Write File [ File:%VibronoDirectory/forecast.json Text:%myjsonresult Append:Off Add Newline:On ]

    So I have a number of questions and pleas:
    1) Is there a way of using AutoTools to read the XML file from the FTP server directly?
    2) Is there a way of handling the forecast period as an array? At the moment I explicitly name each element fperiod0 through fperiod7 which is messy, but I couldn't figure out how to join 2 sections otherwise
    3) The resulting JSON outputfile appears to be malformed, there are bits missing and braces don't line up as I would expect that they might.
    4) My code is a complete hackjob, so any advice to help me introduce a little elegance would be very much appreciated.

    Many thanks in advance for any help
     
  2. joaomgcd

    joaomgcd Administrator Staff Member

    Joined:
    Feb 3, 2015
    Messages:
    9,477
    Likes Received:
    806
    Hi there!
    1. There's not, sorry.
    2. I looked at the data, but couldn't understand what period you mean... As far as I can see the forecast-period fields are already arrays, right?
    3. To write JSON I recommend using the JSON Write action :)
     
  3. jrobbom5

    jrobbom5 New Member

    Joined:
    Dec 12, 2018
    Messages:
    8
    Likes Received:
    0
    Thanks for the reply.

    Firstly, thanks for the tip regarding JSON write, I should have thought of that.

    In the data, there is one array like this:
    -----------------------------
    <forecast-period index="0" start-time-local="2018-12-12T17:00:00+11:00" end-time-local="2018-12-13T00:00:00+11:00" start-time-utc="2018-12-12T06:00:00Z" end-time-utc="2018-12-12T13:00:00Z">
    "); background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; vertical-align: bottom; height: 10px;"><text type="forecast">
    Partly cloudy. Slight (30%) chance of a shower, becoming less likely later this evening. The chance of a thunderstorm. Winds northerly 15 to 20 km/h turning east to northeasterly in the evening.
    </text>
    </forecast-period>

    <forecast-period index="1" start-time-local="2018-12-13T00:00:00+11:00" end-time-local="2018-12-14T00:00:00+11:00" start-time-utc="2018-12-12T13:00:00Z" end-time-utc="2018-12-13T13:00:00Z">
    "); background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; vertical-align: bottom; height: 10px;"><text type="forecast">
    Cloudy. Very high (near 100%) chance of rain. The chance of a thunderstorm. Some heavy falls expected, most likely during the late morning and early afternoon. Winds north to northeasterly 15 to 25 km/h shifting south to southwesterly during the morning then becoming light in the early afternoon.
    </text>
    <text type="fire_danger">Low-Moderate</text>
    "); background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; vertical-align: bottom; height: 10px;"><text type="uv_alert">
    Sun protection 8:50am to 5:40pm, UV Index predicted to reach 13 [Extreme]
    </text>
    </forecast-period>
    -----------------------------

    And then a little further down, there is another array with the same xml path, like this:
    -----------------------------
    <forecast-period index="0" start-time-local="2018-12-12T17:00:00+11:00" end-time-local="2018-12-13T00:00:00+11:00" start-time-utc="2018-12-12T06:00:00Z" end-time-utc="2018-12-12T13:00:00Z">
    <element type="forecast_icon_code">3</element>
    <text type="precis">Partly cloudy.</text>
    <text type="probability_of_precipitation">30%</text>
    </forecast-period>

    <forecast-period index="1" start-time-local="2018-12-13T00:00:00+11:00" end-time-local="2018-12-14T00:00:00+11:00" start-time-utc="2018-12-12T13:00:00Z" end-time-utc="2018-12-13T13:00:00Z">
    <element type="forecast_icon_code">16</element>
    <element type="precipitation_range">30 to 50 mm</element>
    <element type="air_temperature_minimum" units="Celsius">19</element>
    <element type="air_temperature_maximum" units="Celsius">23</element>
    <text type="precis">Rain and storms. Possible heavy falls.</text>
    <text type="probability_of_precipitation">100%</text>
    </forecast-period>
    -----------------------------
    And I am trying to merge the two arrays.
     
  4. joaomgcd

    joaomgcd Administrator Staff Member

    Joined:
    Feb 3, 2015
    Messages:
    9,477
    Likes Received:
    806
    Maybe it's best if we look at the JSON directly :)
    Here's the JSON
    Can you give me an example of what data you want from here?
     
  5. jrobbom5

    jrobbom5 New Member

    Joined:
    Dec 12, 2018
    Messages:
    8
    Likes Received:
    0
    Thanks, I agree it is better to look at the JSON :)

    Near the top of the file, maybe 30 lines from the top, there is a section that look likes this:
    ---------------------------------------------------------
    {
    "aac": "VIC_ME001",
    "description": "Melbourne",
    "type": "metropolitan",
    "parent-aac": "VIC_FA001",
    "forecast-period": [
    {
    "index": 0,
    "start-time-local": "2018-12-13T17:00:00+11:00",
    "end-time-local": "2018-12-14T00:00:00+11:00",
    "start-time-utc": "2018-12-13T06:00:00Z",
    "end-time-utc": "2018-12-13T13:00:00Z",
    "text": {
    "type": "forecast",
    "content": "Cloudy. High (70%) chance of showers. The chance of a thunderstorm. Winds south to southeasterly 15 to 20 km/h becoming light in the evening."
    }
    },
    {
    "index": 1,

    ............. (deleted for brevity)

    ]
    },
    ............. (deleted for brevity)
    ---------------------------------------------------------

    And then a little further down, there is another section with the same JSON Path, product.forecast.area.forecast-period[]
    ---------------------------------------------------------
    {
    "aac": "VIC_PT042",
    "description": "Melbourne",
    "type": "location",
    "parent-aac": "VIC_ME001",
    "forecast-period": [
    {
    "index": 0,
    "start-time-local": "2018-12-13T17:00:00+11:00",

    ............. (deleted for brevity)

    },
    {
    "index": 1,

    ............. (deleted for brevity)

    },
    ............. (deleted for brevity)

    ---------------------------------------------------------

    I am trying to merge those two arrays, and discard everything else.

    Thanks very much for your help, it is much appreciated :)
     
  6. joaomgcd

    joaomgcd Administrator Staff Member

    Joined:
    Feb 3, 2015
    Messages:
    9,477
    Likes Received:
    806
    Check out this one. I think it works :)
    Code (Text):
        Test (15)
            A1: AutoTools Json Read [ Configuration:Input Format: Json
        Simple Mode: true
        Json: https://tasker.joaoapps.com/tests/weather.json
        Fields: forecast-period()
        Filter Fields: index
        Filter Values: >0
        Variable Name: forecast()
        Separator: , Timeout (Seconds):60 ]
            A2: Variable Set [ Name:%seperator To:$$$$$$$ Recurse Variables:Off Do Maths:Off Append:Off ]
            A3: Variable Join [ Name:%forecast Joiner:%seperator Delete Parts:Off ]
            A4: AutoTools Json Write [ Configuration:Separator: %seperator
        Add to Array Key: forecasts
        Add to Array Values: %forecast
        Arrays Separator: | Timeout (Seconds):60 ]
       
    Let me know if it works for you!
     
  7. jrobbom5

    jrobbom5 New Member

    Joined:
    Dec 12, 2018
    Messages:
    8
    Likes Received:
    0
    Wow Joao, that is so much neater, I will give it a try and let you know. Thanks so much for your help :) :)
     
  8. jrobbom5

    jrobbom5 New Member

    Joined:
    Dec 12, 2018
    Messages:
    8
    Likes Received:
    0
    Hi Joao,

    Well the output is certainly a lot better than I had managed to achieve. There's just one problem, and that is that the arrays in the output are not merged, there are still 2 distinct arrays with elements index:0 to index:7. These need to be merged so that there is just one array. I have attached the output.

    I will have a look and see if I can figure it out, but any advice would be much appreciated. Thanks :)
     

    Attached Files:

Share This Page