Django JSONField is not able to encode smileys properly

I plan to store a dict in a Django JSONField. One key of this dict is a comment a user can enter. And users like a lot to add some smileys in their comments...

The problem is that some smileys, are saved properly in DB. The database is MySQL 8.0.31, Django version is 4.0.8 : JSONField is supported for this environment as reported in documentation. The database default encoding is utf8mb4 and collation utf8mb4_general_ci.

With that model :

class TestJSONField(models.Model):
    data = models.JSONField()

Here is the test case :

comment=b'smiley : \xf0\x9f\x98\x8a'.decode()
t=TestJSONField(pk=1, data={'comment':comment})
t.save()
r=TestJSONField.objects.get(pk=1)
print('BEFORE :', comment)
print('AFTER :', r.data['comment'], '(str)')
print('AFTER :', r.data['comment'].encode(), '(utf-8 encoded bytes)')

which gives :

BEFORE : smiley : 😊
AFTER : smiley : ? (str)
AFTER : b'smiley : ?' (utf-8 encoded bytes)

As you can see the smiley is not stored correctly. This smiley is 4 bytes encoded, this may be the source of the problem because with 2-bytes encoded chars I do not have any problem.

With a TextField and using json dumps()/loads() I do not have any problem.

Do you have an idea how to have 4 bytes encoded smileys to be saved in a JSONField ?

Back to Top