C version of `json.dumps` accesses dict internals even for dict subclasses

[aljungberg]

Bug report

Bug description:

The optimised JSON encoder disagrees with the normal JSON encoder when encoding the following dict. The example here is as minimalistic as possible but in reality you could imagine the dict being, for example, a copy on write proxy of another dict or something similarly useful.

import json

class StaticDict(dict):
    'Most minimal dict subclass that still fails'

    def keys(self):
        return ["a", "b"]

    def values(self):
        return [1, 2]

    def items(self):
        return zip(self.keys(), self.values())

    def __len__(self):
        return 2

sd = StaticDict()

c_encoder_version = json.dumps(sd)  # {}

json.encoder.c_make_encoder = None
vanilla_version = json.dumps(sd)  # {"a": 1, "b": 2}

if c_encoder_version != vanilla_version:
    raise Exception(f"JSON C encoder disagrees with regular encoder: {c_encoder_version} != {vanilla_version}")

The culprit looks like this code which blindly assumes that every dict is a regular cpython dict. A related kind of bug affected OrderedDict and was fixed by switching to PyDict_CheckExact in one place. While that was enough to fix OrderedDict, the bug under discussion remains.

Discussion

It has been argued before that json.dump should only handle a very limited set of classes, so for example UserDict might be out of scope. Without taking a stance on which types the module should handle in general, dict certainly is one type json should handle. Since a subclass of a dict is still a dict -- not only does it quack like a duck but it literally is a kind of duck -- it seems reasonable to expect it to function in any context a regular dict does, including for this module. As long as it works like a dict, it should... work like a dict.

Failing that, throwing an exception for dict subclasses would seem the least worst alternative. This would break OrderedDict support but if one takes the json only works with native dicts stance, it should.

CPython versions tested on:

3.11

Operating systems tested on:

macOS

Linked PRs

• gh-110941: Fix json.dump encoding dict subclasses as empty. #111036
• gh-110941: Fix json.dump encoding dict subclasses as empty (continued) #150631