Convert several layers deep nested JSON to instance of a class with nested class attributes

Hello, I’m trying to write tests using pytest library. I have a JSON file that I want to have as an input so I made a pytest.fixture to return the data converted to PlanRequest class instance, because that’s what I need to work with in my test function.

def test_data():
    file = os.path.join(os.path.dirname(__file__), 'test_data.json')
    with open(file) as json_file:
        json_data = json.load(json_file)
        job = PlanRequest(json_data)
        return job
class PlanRequest(Plan):
    def initialize_tasks(self, data, **kwargs):
        data['path'] = [i.init(data['id']) for i in data['path']]
        data['tasks'] = [i.init(data['id']) for i in data['tasks']]
        return data
class Task(DynamoTableStructureBase):
    id: UUID
    cmd: CommandType
    plan_id: UUID = None
    type: TaskType = TaskType.PATH
    params: dict[ParamName, Any] = field(default_factory=dict)
    result: dict[str, Any] = field(default_factory=dict)
    order: int = field(default_factory=lambda: -1)

    hash_key: str = 'plan_id'
    range_key: str = 'id'

class GrowinPlanOperation:
    procedure: TaskCommand
    params: dict[str, Any] = optional_field(
        default=MISSING, default_factory=dict

class GrowinPlanMeta:
    field: str = optional_field()
    field_block: str = optional_field()
    crop: str = optional_field()
    crop_variant: str = optional_field()
    started_at: datetime = optional_field()
    params: dict[str, str] = optional_field(
        default=MISSING, default_factory=lambda: {}
    plant_loc: List[GpsPointRaw] = optional_field(
        default=MISSING, default_factory=list
    operation: List[GrowinPlanOperation] = optional_field(
        default=MISSING, default_factory=list

    def _post_dump(self, data, many, **kwargs):
        if data.get('plant_loc'):
            del data['plant_loc']
        return data

class UpdateLogEntry:
    when: datetime
    status: str

class Plan(DynamoTableStructureBase):
    id: UUID
    serial_number: str = optional_field
    assignment: str = None
    client_account_id: str = None
    start_at: datetime = optional_field(
    path: List[Task] = field(default_factory=list)
    tasks: List[Task] = field(default_factory=list)
    automation: List[CommandType] = field(default_factory=list)
    no_connection_mode: bool = field(default=False)
    status: PlanStatus = optional_field(default=PlanStatus.SCHEDULED)
    last_update_at: datetime = optional_field(default=lambda:
    assigned_at: datetime = optional_field(default=None)
    type: MessageType = optional_field(default=lambda: MessageType.PLAN)
    assignment_robot_message_id: UUID = optional_field(default=None)
    growing_plan: GrowinPlanMeta = optional_field(default=GrowinPlanMeta)
    update_log: List[UpdateLogEntry] = field(default_factory=list)
    percent_done: int = optional_field(default=0)
    message: str = optional_field(default='')

    hash_key: str = 'serial_number'
    range_key: str = 'id'

However, the JSON is several layers deep nested and when I try to convert it to the Plan class instance, only the top level layer gets converted to Plan instance, all the other nested attributes stay as type dict and not the types of classes the Plan dataclass defines, the attribute classes define and so on.

Is there way to convert JSON to an object with all the nested attributes being the correct classes per defined dataclass? Hope this makes sense and thank you for your help

You could define your own from_dict to do the recursive instantiation, or make your classes inherit from pydantic’s BaseModel to then use their from_json, or use dacite’s from_dict.

and if you want something less padantic than Pydantic :slight_smile: – there’s my flexi package:

IT’s not very well documented, but it is robust – I developed it for a project that required very complex nested JSON / Object structure:
(GitHub - NOAA-ORR-ERD/adios_oil_database: System for managing petroleum, product data for use in oil spill response)

One of these days, I’ll make a proper stand along package of it …

In the meantime, think of flexi as a demo – but I suspect it will work for you out of the box if your needs are simple


You can look into my cattrs library, this is one of the main use cases.