Populating an SQLite database from Markdown files in Django

I'm starting to learn Django and Python to migrate my website, and I'm trying to wrap my head around getting an SQLite database set up and populated. All the tutorials I see show you how to do this using the admin application and using the web UI to add posts manually, but I want Django to find the Markdown files in my folder structure and pull the info from those Markdown files and populate the database with that data.

This topic (Populating a SQLite3 database from a .txt file with Python) seems to be in the direction I'm heading, but 1) it's old, and 2) it involves a CSV file. (Does this need to be in CSV? Would JSON work just as well or better, especially since I have a JSON feed on the site and will need to eventually populate a JSON file anyways?) Otherwise, I've found topics about importing files from folders to populate the database, but those tips are for Python in general, and I wasn't sure if Django had specific tools to do this kind of thing just as well or better.

What's a good way to set up Django to populate an SQLite database from Markdown files? Does all the code I'd need for this go in the models.py file, or would I need separate Python scripts and/or text files elsewhere in my project and/or application to run?

Right now my articles are in a folder structure like articles/YYYY/MM/article-name.md. I'm using Eleventy at the moment (it's busted for me, which is why I'm migrating), and so the Markdown files have YAML at the top of each file for things like title, date, tags, etc. that I'll need to figure out how to get Django to read those and populate the database with that data too, and eventually Jinja2 code once I get that configured. The posts usually have images and video files, too, so would I need to insert all this as BLOB data?

I may be jumping the gun a bit with that last paragraph, I'm fine just starting out getting a simple "Hello World" example working and progressing from there.

Let's assume this is your Markdown file.

---
title: "My First Article"
date: 2024-09-07
tags: ["django", "python", "markdown"]
image: "first_article.jpg"
---
# This is the content of my first article.

Hello, world!

First, you need to create a model for articles. (Ensure to install Pillow for ImageField-pip install Pillow)

from django.db import models

class Article(models.Model):
    title = models.CharField(max_length=255)
    date = models.DateField()
    tags = models.CharField(max_length=255)
    content = models.TextField()
    slug = models.SlugField(unique=True)
    image = models.ImageField(upload_to='images/', null=True, blank=True)

    def __str__(self):
        return self.title

Second, you need PyYAML for reading the YAML front matter and markdown for converting Markdown content-pip install PyYAML markdown

This is my demo code to the SQLite database from a Markdown file.

def import_markdown_file(self, file_path):
    with open(file_path, 'r') as f: # file_path is a path to your Markdown file.
        content = f.read()

    # Separate YAML front matter and Markdown content
    front_matter, markdown_content = content.split('---\n', 1)[1].split('---\n', 1)

    # Parse the YAML front matter
    metadata = yaml.safe_load(front_matter)

    # Convert Markdown content to HTML (optional)
    html_content = markdown.markdown(markdown_content)

    # Create a new Article object
    article = Article(
        title=metadata.get('title', 'Untitled'),
        date = metadata.get('date', '2024-01-01'),
        tags=','.join(metadata.get('tags', [])),
        content=html_content if html_content else 'Default content',
        slug=os.path.splitext(os.path.basename(file_path))[0],
    )
    
    # Handle the image
    if 'image' in metadata and metadata['image']:
        image_path = os.path.join(os.path.dirname(file_path), metadata['image'])
        if os.path.exists(image_path):
            with open(image_path, 'rb') as img_file:
                # Save image to the ImageField in the Article model
                article.image.save(os.path.basename(image_path), File(img_file), save=False)

    # Save the article object to the database
    article.save()

FYI, images will be uploaded to the media directory(/media/images) in your project. Add these lines in your settings.py

MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')

I hope this will solve your issue. Let me know in the comments if you need any further assistance.

You can create a fixture JSON file using your markdown, which can be easily loaded to your database using python manage.py loaddata <your_fixture_file_path> command.

Docs:- https://docs.djangoproject.com/en/5.1/topics/db/fixtures/

Format of JSON fixture:-

[
   {
        "model": "appname.ModelName",
        "pk": 1,
        "fields": {
            "field1": "Value1",
            "field2": "Value2"
        }
    },
    ....
]
Back to Top