How do I make a custom field output different datatypes depending on file asking for data?
I'm building a molecule properties displaying website using django and sqlite. The database takes a numpy array and stores it as a BLOB. Currently, it just outputs the numpy array back out when asked for it. I'd like it to check what kind of file is asking the data, and output a numpy array if python is asking for it ( for instance for a data integrity test ), and a json object if requested by a javascript file ( like for molecule visualization ).
I'm not even sure if it's possible, so I guess my question is twofold. One, is it possible? Two, if it's not, as long as the data is read only, are there any major problems that might come up if I let inputs be as numpy arrays, and outputs be as json.
The custom field is defined as so:
The packages used:
import io
import json
import numpy as np
from django.db.models import BinaryField
from django.core.exceptions import ValidationError
from django.utils.translation import gettext_lazy as _
For the current numpy only implementation:
# Stores numpy array as BLOB then returns BLOB as np array
class NpArrayField(BinaryField):
"""
Turns numpy array into raw binary data to store in database. When take from database, it outputs the numpy array.
"""
def from_db_value(self, value, expression, connection):
if value is None:
return None
buffer = io.BytesIO(value)
return np.load(buffer, allow_pickle=True) # returns numpy array
def to_python(self, value):
if value is None or isinstance(value, np.ndarray):
return value
# If it's bytes from DB or form
if isinstance(value, (bytes, bytearray)):
buffer = io.BytesIO(value)
return np.load(buffer, allow_pickle=True)
raise ValidationError(_("NpArrayField value must be a numpy.ndarray or a byte/bytearray"))
def get_prep_value(self, value):
if value is None:
return None
if not isinstance(value, np.ndarray):
raise ValidationError(_("NpArrayField value must be a numpy.ndarray"))
buffer = io.BytesIO()
np.save(buffer, value, allow_pickle=True)
return buffer.getvalue()
def deconstruct(self):
name, path, args, kwargs = super().deconstruct()
return name, path, args, kwargs
An implementation which takes a numpy array (stored as a BLOB), and which outputs json:
class NpArrayEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, numpy.ndarray):
return obj.tolist()
return json.JSONEncoder.default(self,obj)
class NpArrayField(BinaryField):
"""
Turns numpy array into json style raw binary data to store in database. When taken from the database, it outputs the json
"""
def from_db_value(self, value, expression, connection):
if value is None:
return None
return value.decode('utf-8'))
def to_python(self, value):
if value is None:
return None
return value
def get_prep_value(self, value):
if value is None:
return None
if not isinstance(value, np.ndarray):
raise ValidationError(_("NpArrayField value must be a numpy.ndarray"))
np_as_json = json.dumps({"array" : value}, cls=NpArrayEncoder)
return bytes(np_as_json, 'utf-8')
def deconstruct(self):
name, path, args, kwargs = super().deconstruct()
return name, path, args, kwargs
In case anyone was wondering, these arrays should be immutable, so it makes sense in this paradigm to store them as atomic units rather than as individual values.
This seems to me like a separation of concerns problem, and it seems like you are trying to adapt your Model/Field to handle use cases outside of it's concerns.
A Model/Field should only be interested in storing/retrieving the data in a consistent format, regardless of use case.
You mention 2 use cases (integrity testing and presentation) neither of these need to be handled at the Model layer.
The test needs to be adapted to the data format, not the other way around.
When it comes to using the data on the frontend, this is a view concern. Assuming you will be having an API to serve the data to your JS component, that is the best place to change the format of the data before sending to the client.
def molecule_view(request, pk):
mol = Molecule.object.get(id=pk)
return JsonResponse({"response": mol.array.tolist()})