Publish Parts Of Obsidian To My Personal Site


The idea at the start was simple. Do something like Obsidian Publish , so read frontmatter and if it contains published: True put it up on a website to view.

The Basics

I started with taking a look at different static site generators but after a bit of testing I ended up back at Hugo, which I already use for ps1.guru and a few other projects.

First I placed a Hugo project loosely based on the GitLab Pages template in .publish/ to work with.

Selecting Notes To Publish

From there I thought about how to get just a subset of notes into a publishable state.

I solved that with a small Python script utilizing the python-frontmatter library to interact with the frontmatter content that places somewhat processed notes into .publish/content/post/.

"""Extract .md files matching certain criteria into a folder for publishing

Required packages:
- frontmatter
- python-frontmatter
"""
import os
from pathlib import Path

import frontmatter as fm

# All paths assume our current working directory is at the root of the obsidian project
SOURCE_FILES = Path("technologies/")
DESTINATION_FOLDER = Path(".publish/content/post/")


def get_published_notes():
    """Generator yielding paths and content of notes that are marked as published"""
    for item in SOURCE_FILES.rglob("*"):
        if item.is_file():
            with open(item) as f:
                note = fm.load(f)
                if note.metadata.get("published", False):
                    print(f"Publishing {item}")
                    # Find the h1 and remove it from the file
                    content_list = note.content.splitlines()
                    title = ""
                    item_to_remove = None
                    for index, line in enumerate(content_list):
                        if line.startswith("# "):
                            title = line.replace("# ", "")
                            item_to_remove = line
                            break
                    if item_to_remove:
                        content_list.remove(item_to_remove)
                    # If no title has been set so we take the h1 from earlier and set it as title
                    if not note.metadata.get("title"):
                        note.metadata["title"] = title
                    note.content = "\n".join(content_list)

                    yield item, note


def run():
    """Run the script"""
    # Ensure our destination folder exists
    os.makedirs(DESTINATION_FOLDER, exist_ok=True)

    for source_path, note in get_published_notes():
        # Construct the final location for the file in the destination
        destination_file = DESTINATION_FOLDER / str(source_path)

        # Create the destination folder recursively if it is missing
        destination_folder = "/".join(str(destination_file).split("/")[:-1])
        os.makedirs(destination_folder, exist_ok=True)

        # Dump the processed note to the destination
        with open(destination_file, "w+") as destination:
            destination.write(fm.dumps(note))


if __name__ == "__main__":
    run()

This script now lives as .note_publish.py in my Obsidian directory, the dot is a concious choice to hide it from the file explorer in Obsidian.

Some assumptions are made in the script:

  • Content is designated as published with published: true in fromtmatter
  • Content has one and only one h1 markdown heading formatted like this # TEXT
  • Content is located in technology/ - This may be the most specific to me thing in the whole script

As I want to run the script inside GitLab CI/CD I added .publish/content/post/ to my .gitignore to never accidentally commit generated content into the repository when testing locally.

Publishing To GitLab Pages

The finishing touch was to bind it all together with GitLab CI/CD, that looks like this.

stages:
  - build
  - pages

variables:
  GIT_SUBMODULE_STRATEGY: recursive

collect_notes:
  stage: build
  image: python:3.9.6-slim
  script:
    - pip install -U frontmatter
    - pip install -U python-frontmatter
    - python .note_publish.py
  artifacts:
    paths:
      - .publish/content/post
  only:
    - master

pages:
  stage: pages
  dependencies:
    - collect_notes
  image: registry.gitlab.com/pages/hugo:latest
  script:
    - cd .publish
    - hugo
    - ls -la
    - mv public ../
  artifacts:
    paths:
      - public
  only:
    - master

Some interesting things to note here:

  • The hugo:latest image used to build the static site does not play well with python, therefore I use a dependency job to first run the custom script I wrote in a Python container and pass the generated files over using artifacts.
  • GIT_SUBMODULE_STRATEGY: recursive is required because I have my theme in a submodule

With all of this I just needed to wire up a domain, marco.kamner.eu in this case, and let GitLab Pages work its magic.

Adding Some Static Pages

Now that the dynamic parts are done I just added some static pages (About & Portfolio) directly to the Hugo content directory and linked them in the menu.

Resume

I honestly thought that this would be much much easier than it turned out to be. But after all I think this setup will allow me to enjoy a automated system that enables me to share my knowledge faster and without almost any manual work.

If you wan’t to use this setup too feel free to copy the instructions from above or shoot me a quick message over on Twitter and I can create a template project on GitLab.

See also