Django.fun

How to convert Django Management Commands into Form Actions to populate database with an uploaded CSV file?

Django 1.9.8, Python 3.6, Postgres database

I am writing a small Django app that will use the UI to import products from an uploaded csv file via a Form and populate the database. I have written the backend logic for the app using Custom Management Commands but am struggling confused about how to connect this to the frontend using Forms.

Here's an example of the csv file:

name,sku,description
Brian James,skus-will-look-like-this,The products will have various descriptions. And multiple lines too.

I have the following logic implemented:

  1. Read a large csv file of 500K products to the database using SKU as a unique field.
  2. When the file has uploaded, a live stream of what is happening should be displayed. print() currently serves this purpose but I think it should be possible to implement this with SSE on the front end.
  3. All products can be searched / filtered.
  4. Be able to delete all existing records and start a fresh upload as well as being able to add/update products manually.

I created a simple form uploader as a separate project to play around with it, but even after this and reading the docs I am not clear on how to make the jump from "converting" the management commands into viable actions through Forms.

Here is the relevant code:

This uploads a local csv file, parses it, updates only specific fields if SKU already exists, creates 'active/inactive' status, and prints after each save so user can see what is happening.

/products/management/commands/import_products.py

class Command(BaseCommand):

    def handle(self, *args, **options):

        print('Importing data from:', settings.DATA_IMPORT_LOCATION)

        with open(f'{settings.DATA_IMPORT_LOCATION}/products.csv', 'r') as products_csv:
            products_file = csv.reader(products_csv)
            next(products_file)  # skip header row

            for counter, line in enumerate(products_file):
                name = line[0]
                sku = line[1]
                description = line[2]

                # Check if sku already exists if this just needs to be updated
                if Product.objects.filter(sku=sku).exists():
                    sku_id = Product.objects.only('id').get(sku=sku).id
                    p = Product(id=sku_id)
                    p.name = name
                    p.description = description
                    p.status = random.choice(['active', 'inactive'])
                    p.save(update_fields=['name', 'description', 'status'])
                    print(p)

                else:
                    p = Product()
                    p.name = name
                    p.sku = sku
                    p.description = description
                    p.status = random.choice(['active', 'inactive'])
                    p.save()
                    print(p)

            print(f'Import complete, imported {counter} products')

This deletes all objects from the database /products/management/commands/delete_products.py

from products.models import Product
from django.core.management.base import BaseCommand

class Command(BaseCommand):
    def handle(self, *args, **options):
        Product.objects.all().delete()

models.py

class Product(models.Model):
    name = models.TextField(blank=False, null=False)
    sku = models.TextField(blank=False, null=False, unique=True)
    description = models.TextField(blank=False, null=False)
    status = models.TextField(blank=False, null=False, default='inactive')

    def __str__(self):
        return [self.name, self.sku, self.description, self.status]

I should be able to do something like this:

Add document = models.FileField(upload_to='documents/') to models.py

Add forms.py

from django import forms
from models import Product

class ProductForm(forms.ModelForm):
    class Meta:
        model = Product
        fields = ('name', 'sku', 'description', 'status', 'document')

/templates/products/model_form_upload.html

{% extends 'base.html' %}

{% block content %}
  <form method="post" enctype="multipart/form-data">
    {% csrf_token %}
    {{ form.as_p }}
    <button type="submit">Upload</button>
  </form>

  <p><a href="{% url 'home' %}">Return home</a></p>
{% endblock %}

/templates/products/home.html

{% extends 'base.html' %}

{% block content %}
  <ul>
    <li>
      <a href="{% url 'model_form_upload' %}">Model Form Upload</a>
    </li>
  </ul>

  <p>Uploaded files:</p>
  <ul>
    {% for obj in products %}
      <li>
        <a href="{{ obj.product.url }}">{{ obj.product.name }}</a>
        <small>(Uploading: {{ obj.name }})</small>
      </li>
    {% endfor %}
  </ul>
{% endblock %}

Adding this to views.py:

def home(request):
    products = Product.objects.all()
    return render(request, 'products/home.html', { 'products': products })

def model_form_upload(request):
    if request.method == 'POST':
        form = ProductForm(request.POST, request.FILES)
        if form.is_valid():
            form.save()
            return redirect('home')
    else:
        form = ProductForm()
    return render(request, 'products/model_form_upload.html', {
        'form': form
    })

Adding to urls.py:

urlpatterns = [
url(r'^$', views.home, name='home'),
url(r'^uploads/form/$', views.model_form_upload, name='model_form_upload'),
]

And adding this to settings.py:

STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')

Adding this code crashes the app and results in multiple errors (improperly configured urls, the page not loading, the data not being saved, etc). I'm not sure how to integrate the Form with the logic from the Management Commands so that the csv file uploaded through the Form actually gets 'grabbed' and parsed as happens in import_products.py -> with open(f'{settings.DATA_IMPORT_LOCATION}/products.csv', 'r') as products_csv:

Would appreciate any help or being directed to some sources that examine this workflow of CRUD operations with Forms.