Help with calling data from JSON

There is a project I was trying to follow on Github that was last modified many years ago and was then updated a year or so after that, but again, even that was several years ago. I cannot reach either of the previous two content creators for help, making me assume that they are no longer interested in the project. The project revolves around querying a JSON file from an OpenSprinkler device and manipulating the data to display on a LCD. I’ve found many areas of the code designed by others that I could fix but I am having a big failure with importing and parsing the JSON data into variables the Python code can work with.

{"settings":{"devt":1717414572,"nbrd":1,"en":1,"sn1":0,"sn2":0,"rd":0,"rdst":0,"sunrise":342,"sunset":1227,"eip":2728279896,"lwc":1717406493,"lswc":1717406493,"lupt":0,"lrbtc":5,"lrun":[1,2,4050,1717397821],"pq":0,"pt":0,"nq":0,"mac":"B8:27:EB:3E:CF:85","loc":"38.79776,-121.36498","jsp":"https://ui.opensprinkler.com/js","wsp":"weather.opensprinkler.com","wto":{"baseETo":0.19,"elevation":600,"pws":"","key":""},"ifkey":"","mqtt":{"en":1,"host":"172.19.20.88","port":1883,"user":"mqtt-user","pass":"testing"},"wtdata":{"wp":"Apple","eto":0.287,"radiation":9.54,"minT":59,"maxT":90,"minH":26,"maxH":79,"wind":4.3,"p":0},"wterr":0,"dname":"My OpenSprinkler","sbits":[0,0],"ps":[[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0]],"gpio":[5,6,7,8,9,10,11,12,13,16,18,19,20,21,23,24,25,26]},"programs":{"nprogs":3,"nboards":1,"mnp":40,"mnst":4,"pnsize":32,"pd":[[51,0,2,[20540,0,0,0],[2700,0,0,0,0,0,0,0],"Front Yard",[0,33,415]],[3,127,0,[20495,0,0,0],[0,2700,0,0,0,0,0,0],"Garden Morning",[0,33,415]],[3,127,0,[930,0,0,0],[0,2400,0,0,0,0,0,0],"Garden Afternoon",[0,33,415]]]},"options":{"fwv":220,"tz":20,"hp0":144,"hp1":31,"hwv":64,"ext":0,"sdt":0,"mas":0,"mton":0,"mtof":0,"wl":150,"den":1,"ipas":0,"devid":0,"uwt":3,"ntp1":0,"ntp2":0,"ntp3":0,"ntp4":0,"lg":1,"mas2":0,"mton2":0,"mtof2":0,"fwm":3,"fpr0":100,"fpr1":0,"re":0,"sar":0,"ife":0,"sn1t":0,"sn1o":1,"sn2t":0,"sn2o":1,"sn1on":0,"sn1of":0,"sn2on":0,"sn2of":0,"wimod":169,"reset":0,"dexp":-1,"mexp":24,"hwt":255,"ms":[0,120,120,0,120,120]},"status":{"sn":[0,0,0,0,0,0,0,0],"nstations":8},"stations":{"masop":[255],"masop2":[0],"ignore_rain":[0],"ignore_sn1":[0],"ignore_sn2":[0],"stn_dis":[252],"stn_spe":[0],"stn_grp":[0,0,0,0,0,0,0,0],"snames":["Front of House","Backyard/Garden","S03","S04","S05","S06","S07","S08"],"maxlen":32}}

By looking through the JSON, you might be able to see one problem, there is a field name called “pass”. The field name is for storing an MQTT password. I have tried to distill the entire Python code down to the part where I believe I am having issues.

#!/usr/bin/env python


import json
import locale
from collections import namedtuple
from time import *
from urllib.request import urlopen
from subprocess import check_output

""" ################ Parameters #################### """
osAddress = "127.0.0.1"  
osPort = 8080  
md5hash = "a6d82bced638de3def1e9bbb4983225c"  

################################################
api_url = ("http://"+osAddress+":"+str(osPort)+"/ja?pw="+md5hash)


def get_data(url):
	data = urlopen(url).read()
	data = data.replace("\"pass\":", "\"password\":") # Replace protected keyword "pass" with acceptable term "password"
	variables = json.loads(data, object_hook=lambda d: namedtuple('X', d.keys())(*d.values()))
	return variables

# Parse JSON into an object with attributes corresponding to dict keys.
ja = get_data(api_url)

print(ja)

I’ve removed a lot of code that I don’t think is relevant and that last line of the code is my way of debugging. If I run the code as is, I get:

bigron@OSPi:~$./ospiTEST.py 
Traceback (most recent call last):
  File "/home/bigron/./ospiTEST.py", line 27, in <module>
    ja = get_data(api_url)
  File "/home/bigron/./ospiTEST.py", line 22, in get_data
    data = data.replace("\"pass\":", "\"password\":") # Replace protected keyword "pass" with acceptable term "password"
TypeError: a bytes-like object is required, not 'str'

This is essentially the same error I’m stuck with with the non-distilled down version, so I’m pretty sure I’ve isolated the part I’m having issues with. If I were to, for example, type in the URL created by api_url into a browser or using curl from a shell, it returns the JSON information from the OpenSprinkler controller. As mentioned before, the JSON includes a field name of “pass”, which needs to be renamed “password”. That is why there is a line in the get_data function to search for and replace it. I must admit that I am not sure what is happening in the line that includes json.loads in it, but my assumption is that somehow, when the JSON is acquired, it is being treated as if the whole thing is a long string and not data. At this point, it was copy & pasted without full understanding and my guess is that it stopped working because it was originally designed for something that has been deprecated.If someone could point out my errors and direct me in the right direction, I would appreciate it.

Close, but you haven’t quite removed enough :wink:

The problem here has nothing to do with JSON, as it occurs before your code can get anywhere near that part. The problem is that urlopen(url).read() gives you raw bytes of data, not a string. You need to convert this to string yourself, by specifying an encoding to .decode with - just as you would with any other bytes object.

This sort of detail is why even the standard library documentation suggests using Requests - a friendlier, third-party wrapper - instead.

Aside:

I don’t see why you expect this to be a problem. It sounds like you’re hoping to assign the contents of the parsed JSON data to separate variables, by dynamically using the keys in the dictionary to create variable names. Trust me, you don’t actually want to do this. It adds complexity for no benefit. Just use the data structure that’s actually given to you. There’s nothing preventing Python from using the string "pass" as a dictionary key (and if there were, it would be too late for your code to do anything about it).

1 Like