Convert several layers deep nested JSON to instance of a class with nested class attributes

Python Help
1

Hello, I’m trying to write tests using pytest library. I have a JSON file that I want to have as an input so I made a pytest.fixture to return the data converted to PlanRequest class instance, because that’s what I need to work with in my test function.

@pytest.fixture
def test_data():
    file = os.path.join(os.path.dirname(__file__), 'test_data.json')
    with open(file) as json_file:
        json_data = json.load(json_file)
        job = PlanRequest(json_data)
        _preprocess_job(job)
        return job

@dataclass
class PlanRequest(Plan):
    @post_load
    def initialize_tasks(self, data, **kwargs):
        data['path'] = [i.init(data['id']) for i in data['path']]
        data['tasks'] = [i.init(data['id']) for i in data['tasks']]
        return data

@dataclass
class Task(DynamoTableStructureBase):
    id: UUID
    cmd: CommandType
    plan_id: UUID = None
    type: TaskType = TaskType.PATH
    params: dict[ParamName, Any] = field(default_factory=dict)
    result: dict[str, Any] = field(default_factory=dict)
    order: int = field(default_factory=lambda: -1)

    hash_key: str = 'plan_id'
    range_key: str = 'id'


@dataclass
class GrowinPlanOperation:
    procedure: TaskCommand
    params: dict[str, Any] = optional_field(
        default=MISSING, default_factory=dict
    )


@dataclass
class GrowinPlanMeta:
    field: str = optional_field()
    field_block: str = optional_field()
    crop: str = optional_field()
    crop_variant: str = optional_field()
    started_at: datetime = optional_field()
    params: dict[str, str] = optional_field(
        default=MISSING, default_factory=lambda: {}
    )
    plant_loc: List[GpsPointRaw] = optional_field(
        default=MISSING, default_factory=list
    )
    operation: List[GrowinPlanOperation] = optional_field(
        default=MISSING, default_factory=list
    )

    @post_dump(pass_many=True)
    def _post_dump(self, data, many, **kwargs):
        if data.get('plant_loc'):
            del data['plant_loc']
        return data


@dataclass
class UpdateLogEntry:
    when: datetime
    status: str


@dataclass
class Plan(DynamoTableStructureBase):
    id: UUID
    serial_number: str = optional_field
    assignment: str = None
    client_account_id: str = None
    start_at: datetime = optional_field(
        default=MISSING, default_factory=datetime.now()
    )
    path: List[Task] = field(default_factory=list)
    tasks: List[Task] = field(default_factory=list)
    automation: List[CommandType] = field(default_factory=list)
    no_connection_mode: bool = field(default=False)
    status: PlanStatus = optional_field(default=PlanStatus.SCHEDULED)
    last_update_at: datetime = optional_field(default=lambda: datetime.now())
    assigned_at: datetime = optional_field(default=None)
    type: MessageType = optional_field(default=lambda: MessageType.PLAN)
    assignment_robot_message_id: UUID = optional_field(default=None)
    growing_plan: GrowinPlanMeta = optional_field(default=GrowinPlanMeta)
    update_log: List[UpdateLogEntry] = field(default_factory=list)
    percent_done: int = optional_field(default=0)
    message: str = optional_field(default='')

    hash_key: str = 'serial_number'
    range_key: str = 'id'

However, the JSON is several layers deep nested and when I try to convert it to the Plan class instance, only the top level layer gets converted to Plan instance, all the other nested attributes stay as type dict and not the types of classes the Plan dataclass defines, the attribute classes define and so on.

Is there way to convert JSON to an object with all the nested attributes being the correct classes per defined dataclass? Hope this makes sense and thank you for your help

2

You could define your own from_dict to do the recursive instantiation, or make your classes inherit from pydantic’s BaseModel to then use their from_json, or use dacite’s from_dict.

3

and if you want something less padantic than Pydantic :slight_smile: – there’s my flexi package:

IT’s not very well documented, but it is robust – I developed it for a project that required very complex nested JSON / Object structure:
(GitHub - NOAA-ORR-ERD/adios_oil_database: System for managing petroleum, product data for use in oil spill response)

One of these days, I’ll make a proper stand along package of it …

In the meantime, think of flexi as a demo – but I suspect it will work for you out of the box if your needs are simple

-CHB

4

You can look into my cattrs library, this is one of the main use cases.