Blog in Emacs - Use Jekyll's Draft Mode

Why?

I wasn’t aware of Jekyll’s draft mode. My workaround was manually changing the published field in the front matter to true when the post is ready to publish. It works fine. However, with naive support from Jekyll, there are more benefits to using the draft mode.

To start with, I like the drafts saved in the _drafts folder, not mixed with other published posts in the _posts folder. It is way more cleaner and easy to manage. With a glimpse of my eyes, I can see what are the posts that I am drafting.

It also gives a piece of mind: only posts under the _posts folder are exported and shown in my blog. It ensures I don’t accidentally publish a post in draft.

Once there are files in _drafts folder, adding ​-​-​drafts argument to the jekyll serve command is all I need to be able to see the drafts locally.

Of course, I also need to write a bit of Lisp code to integrate the draft mode into my blogging workflow. This is the remaining of this post is about.

Implementation

For a blog post, I have the source file in org-mode and its exported file in Markdown. Now there is a new location dimension: they can be either in the _drafts or _posts folder.

In terms of content, the published post and its final draft, and their exported counterparts are the same, only in different locations. Their content can be different to have some flexibility, e.g. published post has higher resolution of screenshots. This feature is possible to implement in the future. For now, I follow the simple “same but in different places” rule.

The new process looks like this: When I publish a post, it moves the org file from _draft to _posts folder, adds a date to the filename (which I have already), and then triggers the exporting process. To avoid duplication, it removes the original org file and its exported draft in Markdown.

To achieve that, the main missing piece from my current Emacs configuration is the yt/jekyll-find-export function (see below). For a post in _drafts or _posts, it finds the full path of the corresponding exported markdown file. I can then delete it or start the exporting process.

 
(defun yt/jekyll-is-draft-p ()
  "if the file is inside of the draft directory, it is a draft."
  (let ((draft-dir  (file-truename jekyll-source-drafts-dir))
        (filepath (file-truename (buffer-file-name))))
    (string-prefix-p draft-dir filepath)))


(defun yt/jekyll-find-export ()
  "find the full path to the exported file of the current post."
  (let* ((src-file (file-name-nondirectory (buffer-file-name)))
         (dest-file (file-name-with-extension src-file ".md")))
    (if (yt/jekyll-is-draft-p)
        (file-name-concat jekyll-site-draft-dir dest-file)
      (file-name-concat jekyll-site-post-dir dest-file))))

Blog in Emacs - Work with Images

I do my best to keep my blog simple, I would not use images/videos unless I can’t demonstrate well enough in plain text, for example, to demonstrate a mobile app using a screenshot (Learn in Emacs - Building Up Vocabulary) or how to represent stock price charts for neutral network (Speed Up Sparse Boolean Data).

Even when I do, I keep the usage to the bare minimum: all I do is insert the image, make it centralised, and put a caption on top of it.

The Org-mode supports images well, with a few additional HTML attributes for each inserted image, I can fine-control the images’ position, alignment, size etc.

However, I can’t get the benefits because I migrated my blog posts from HTML to the Markdown format for its simplicity. Plus Jekyll comes with its little quirks when it comes to Markdown images. So I have to write something for myself.

I managed to achieve a satisfactory workflow for my simple usage of images in Emacs. It works well for the Jekyll site. Here’s the code and explanation.

• org-download : is the package that I use to create the images for blogging from various sources.

  I can drag images from external applications to Emacs, including browsers, Preview, or iPhoto. The images will be saved in the /project/assets/org-download folder per my matrix/project setup.

  For the application that I can’t drag the images, I take a screenshot inside Emacs by calling the org-download-screenshot function.

• yt/jekyll-copy-from-org-downkload: is a little helper function that transfers the files under the org-download folder to the /assets folder in a Jekyll site.

  It lists the files in the source org-download folder and provides them as a selection list. It comes with auto-completion and fuzzy matches to help me choose the file.

  It also strips out the special characters in the filename otherwise the URL will be broken in Jekyll.

• yt/jekyll-insert-image: lists the files in the /assets folder so I can choose easily which image to use.

  It brings up the Liquid template for image so I don’t have to remember its syntax. It ensures the file path is in the correct format (starts with ​/assets​/), I just fill in the caption and size after selecting the file.

An extract of the code is listed below for demonstration propose. Future updates will be reflected in my .emacs.d git repo.

 

(defun yt/jekyll-insert-image (src caption)
  (interactive (list (read-file-name "images to include: " jekyll-assets-dir)
                     (read-string "Caption: ")))
  (insert (format jekyll-insert-image-liquid-template (file-name-nondirectory src) caption)))

(defun yt/jekyll-copy-org-download-to-assets (file)
  "copy file from project org-download folder to the blog assets folder.
it ensures there's no underscore(_) in the file name.
"
  (interactive (list (read-file-name "file to copy: " org-download-image-dir)))
  (let* ((ext (file-name-extension file ))
        (base (file-name-base file))
        (dest-base (jekyll-make-slug base))
        (dest-file (expand-file-name (file-name-with-extension dest-base ext) jekyll-site-assets-dir)))
    (copy-file file dest-file)
    dest-file))

Learn in Emacs - Building Up Vocabulary

Table of Contents

1. WHY?
2. Workflow for Building Vocabulary
3. Revision on Mobile Devices
4. Org-mode Based Simple Study Strategies
5. Emacs Lisp Implementation

WHY?

Research shows having effective and rapid communication can boost creativity and spark joy¹. I believe in it from my personal experience in conversing, reading a book in my native language or trying to understand a large codebase.

I wasn’t enable to achieve similar results when it came to using English. In the past I have been trying to improve my English language skills to boost my productivity in reading books and to make it more enjoyable. The approach was practising more in reading and writing. However, I started questioning the effectiveness. This year I decided to take one step back to focus on the basics and improve my vocabulary.

I want to take the “slip-box” method² which proved to be effective for learning Emacs Lisp language. It is a bottom-up approach, so I would have one note for each word with the explanation in it, links to other similar words, or words I got confused with.

One advantage is that I can also leverage my existing setup.

Workflow for Building Vocabulary

When come across a new word that I’m not sure about its meaning, I will

1. move the cursor to the word,
2. press F1 d to look into the dictionary, the result will shown in the osx-dictionary buffer,
3. read its meaning and try to understand it,
4. press r to listen the pronunciation and read after it. I usually repeat it a couple of times to deepen the memory,
5. press a to create an atomic note. it has the dictionary meaning in it for future reference,
6. edit the notes to add my understanding and copy the sentence/paragraph that contains the new word.
7. press C-c C-c to save it to my vocabulary database, which is just a folder with flat org-mode files.

There’s quite a lot of automation so I can focus on understanding it (Step 3) and write a good note (Step 6) in my own words.

This workflow depends on two Emacs packages:

• osx-dictionary: it interfaces with macOS’s dictionary app. It displays the meaning and says the pronunciation.

  The package is well written and easy to work with; I managed to extend it to add Steps 5-7 with little effort.

  It has limitations: it works only in macOS and it only outputs one dictionary. Adding the meaning in Chinese requires a few more manual steps: 1) press o to open the Dictionary.app, 2) go to the Chinese dictionary tab and copy the meaning, and 3) paste it to the note in Emacs.

  I personally find the Oxford dictionary macOS uses is not easy to follow. From time to time I have to visit https://dictionary.cambridge.org/ to find the explanation that I could understand. In transforming my old vocabulary notes to the new format, I found the explanation from vocabulary.com is the best. I might have to resurrect my voca-builder³ package.

• org-roam: it interfaces with org-mode for creating atomic notes. It avoids duplication: if there’s a note for the word that exists already, it opens the note, so I can have a look and enrich it.

  I can link notes/words in my vocabulary database which is very useful because for me learning by comparing is super effective.

  The org-mode provides a lot of functionalities that might be useful to facility learning in the future.

Revision on Mobile Devices

Once I have a fleet of notes, the next step is to revise them on a regularly. The routine I’m trying to get myself into is rereading the notes I created for the last few days while waiting for the tube/bus, I call it a revision break.

So far I have an Emacs lisp program⁴ that filters all my notes by time so I have last_24_hours.org, last_3_days.org and last_7_days.org. These files are synced with iCloud so they are available to review on my iPad and iPhone using the beorg App.

Reading my vocabulary notes on iPhone

Org-mode Based Simple Study Strategies

For the dedicated study sessions, I need a few strategies to shortlist the notes. I think They will be based on the metadata of the note. With org-mode’s API, it should be easy to implement.

I haven’t done it yet, but the idea is to score the notes from 0 to 5, 5 means the most important notes so I would study them first, 0 means not important notes so will be at the bottom.

There can be multiple scores, for example, one for pronunciation. the word that I got the pronunciation completely wrong would get a 5, and the word would get a 3 if sometimes I got it wrong, and sometimes I got it right.

Another score is how many times I looked into the word. There are words that I just keep forgetting about it, or keep confusing with another similar word. So the property of ‘visited_at’ gets a timestamp appended at the time of visiting and the score is calculated by the number of timestamps.

Emacs Lisp Implementation

Adding an action to the headline in osx-dictionary’s buffer.

 
(require 'osx-dictionary)
(setq osx-dictionary-mode-header-line
      (append '((:propertize "a" face mode-line-buffer-id)
                ": Add to vocabulary"
                "    ")
              osx-dictionary-mode-header-line))

Adding yt/add-to-vocabulary to key a in osx-dictionary buffer. It creates an note using org-roam.

 
(defvar vocabulary-repo-dir "~/matrix/learning/meta-leanring/vocabulary"
  "where to save the vocabulary notes.")
(defvar yt/voca--roam-template
  '(("d" "default" plain "%?" :target
     (file+head "%<%Y%m%d%H%M%S>-${slug}.org" "#+title: ${title}

%?

#+begin_example
%(yt/osx-dict--get-meaning)
#+end_example

")
     :unnarrowed t))
  "roam template for vocabulary notes")


(defun yt/add-to-vocabulary ()
  "add a new vocabulary note for ther highlihted region or word at
point."
  (interactive)
  (let* ((org-roam-directory (expand-file-name "notes" vocabulary-repo-dir))
         (org-roam-db-location (expand-file-name "org-roam.db" org-roam-directory ))
         (org-roam-capture-templates yt/voca--roam-template))
    (org-roam-node-find nil (yt/osx-dict--get-word-and-pronounce))))

(defun yt/osx-dict--get-word-and-pronounce ()
  "extract the word and its pronunciation from the *osx-dictionary* buffer"
  (with-current-buffer "*osx-dictionary*"
    (goto-char (point-min))
    (search-forward "|" nil nil 2)
    (buffer-substring-no-properties (point-min) (point))))

(defun yt/osx-dict--get-meaning ()
  "wrap the *osx-dictionary* buffer cnotent as a string"
  (with-current-buffer "*osx-dictionary*"
    (buffer-substring-no-properties (point-min) (point-max))))

(define-key osx-dictionary-mode-map "a" 'yt/add-to-vocabulary)

Footnotes

¹ reference is lost; it is somewhere in the book “The Second Mountain”, the chapter on religions.

² from the book “How to Take Smart Notes”. I plan to reread this book in early 2024.

³ My first Emacs package in 2015, https://github.com/yitang/voca-builder

⁴ next blog post is on my lisp programs

Atomic Habit in Emacs - Keep Git Repos Clean

Table of Contents

1. Why?
2. Emacs Lisp Helper
3. Practise

Why?

I am having a hard time keeping my git repositories clean: there are just too many repositories, I counted 31 in total, and I have 5 computers where I work on them.

The consequence is that sometimes I get surprised at seeing a lot of seemingly useful changes that are not committed to the git repo. I had to stop whatever I was doing to just think about what to do with those changes. It breaks the flow!

There are other occasions where I thought I fixed some bugs, but I don’t have the patches on my laptop. It turned out I didn’t check in to the cloud, so I have to log back to the right server to run a couple of git commands, or if I don’t have access to the servers, I have to fix the bugs from scratch again. It is inefficient!

It can happen a lot in active projects where I work on multiple systems and multiple git repos or when I travel. I plan to revisit my filesystem (which is inspired by Stephen Wolfram ¹) and tech setup to reduce the number of repos by merging them and keeping only 1 laptop, 1 workstation and 1 server. This is something for summer, it can reduce the severity of the problem but can not eliminate it.

At the moment, I just have to become more disciplined in managing files, e.g. to have an atomic habit of checking my git repo regularly, or at least do it once at the end of the day, or as part of the shutdown ritual after finishing a task².

Emacs Lisp Helper

The 3rd Law of Behavior Change is make it easy.

James Clear, Atomic Habit

To facilitate the forming of this habit, I implemented a utility function in Lisp to list the dirty git repo, and provide a clickable link to the magit-status buffer of the git repo. With one click on the hyperlink, I can start to run git commands via the mighty magit package. I bind this action to keystroke F9-G.

 
(defun yt/git--find-unclean-repo (root-dir)
  ""
  ;; (interactive)
  (setq out nil)
  (dolist (dir (directory-files-recursively root-dir "\\.git$" t))
    (message "checking repo %s" dir)
    (let* ((git-dir (file-name-parent-directory dir))
           (default-directory git-dir))
      (unless (string= "" (shell-command-to-string "git status --porcelain"))
        (push git-dir out))))
  out)


(defun yt/dirty-git-repos (&optional root-dir)
  "list the dirty git repos, provides a clickable link to their
magit-status buffer."
  (interactive (list (read-directory-name "Where's the root directory?" )))

  (let ((buffer (get-buffer-create "*test-git-clean*"))
        (git-repos (yt/git--find-unclean-repo root-dir)))
    (with-current-buffer  buffer
      (unless (eq major-mode 'org-mode)
        (org-mode))
      (goto-char (point-min))
      (insert (format "Number of dirty git repos: %s " (length git-repos)))
      (dolist (git-repo git-repos)
        (insert (format "\n[[elisp:(magit-status \"%s\")][%s]]" git-repo git-repo))))
    ))

The workhorse is the git status --porcelain command: If the git repo is clean, it returns nothing, otherwise, it outputs the file names whose changes are not checked in, e.g. the first file is modified (M), and the second file is not untracked (??).

 M config/Dev-R.el
?? snippets/org-mode/metric

The rest of the code is for parsing the outputs and turning them into a user-friendly format in Org-mode. What’s interesting is that The org-mode provides a kind of hyperlink that evaluates Lisp expressions, using the example below,

 
[elisp:(magit-status "/foo")]["Git Status of Repo /foo"]

The description of the hyperlink is “Git Status of Repo /foo” , after I click it, it runs the expression (magit-status "/foo") which shows the git status of /foo repo in a dedicated buffer.

Before executing it will ask for a confirmation. It can be a bit annoying and inconvenienced at first which naturally leads to the temptation of removing this behaviour by setting org-link-elisp-confirm-function to nil. I discourage you from doing so in case someone embeds funny codes, (for example rm -rf ~/) in a hyperlink, so make sure to check that variable’s documentation before changing it³!

Practise

It was fun to write the lisp functions. I learnt how to use the optional function argument and interactive so that the function can be used both interactively and pragmatically. I’m very much wanting to spend more time in coding, to enhance it with some ideas I got from reading Xu Chunyang’s osx-dictionary package⁴.

However, the effectiveness of those functions has little to do with the extra features I had in mind but really depends on how I use them. Solving the problems requires deliberate practise and changing my behaviours so that cleaning git repos becomes a habit of mine, which is always the hardest part.

One key indicator for this habit⁵ can be the number of check-ins and see if there’s a substantial increase from today.

Footnotes

¹ see Stephen Wolfram’s blog posts

² Cal Newport, Deep Work, Page 151

³ https://orgmode.org/manual/Code-Evaluation-Security.html

⁴ https://github.com/xuchunyang/osx-dictionary.el

⁵ inspired by Andrew Grove’s book High Output Management

GPG in Emacs - Functions to Decrypt and Delete All

Table of Contents

1. Motivation
2. Emacs Lisp Implementation
3. Bash Implementation

Motivation

Continuing from my last post, the EPA provides a seamless interface when working with GPG files in Emacs. But there are situations where I have to work with GPG files using other programs (mostly Python) which EPA cannot help.

For those cases, I have to decrypt the GPG files first before using them (for example, calling pandas.read_csv).

Obviously, there’s no point in encrypting a file if there is a decrypted version next to it. So I also need a function to delete all the decrypted files.

Emacs Lisp Implementation

Of course, I run Python inside of Emacs, I wrote the Lisp functions to decrypt GPG files and delete all the decrypted files.

 
(defun yt/gpg--decrypt-recursively (root-dir)
  "It decrypts all the files ends .gpg under the root-dir. The decrypted files have the same filename but without the .gpg extension.

It stops if the decryption fails. 
"
  (interactive)
  (dolist (file (directory-files-recursively root-dir "\\.gpg"))
    ;; the 2nd argument for epa-decrypt-file can only be the base filename without the directory.
    (let ((default-directory (file-name-directory file)))
      (epa-decrypt-file file (file-name-base file))
    ))
  )

(defun yt/gpg--delete-decrypted-files (root-dir)
  "It deletes the decrypted files under the root-dir directory.

e.g. if there's a file foo.tar.gz.gpg, it attempts to remove the foo.tar.gz file.
"
  (interactive)
  (dolist (file (directory-files-recursively root-dir "\\.gpg"))
    (delete-file (file-name-sans-extension file))
    )
  )

A bit of explanation:

• directory-files-recursively: searches for files with a pattern. Here, it returns all the files ending with .gpg under the given root-dir,
• dolist: loops over the GPG files to process them one by one,
• epa-decrypt-file: decrypts a GPG file into a new file.
• delete-file: deletes a given filename.

It seems the epa-decrypt-file function does not like the new filename with the directory in its path, so I have to set the default directory (working directory) and use the base filename after removing the directory as a workaround.

Bash Implementation

It would be useful to have those functionalities outside of the Emacs, so I implemented their counterpart in Bash.

 
function decrypt_recursively() {
    # PS: this function is equivalent to `gpg --decrypt-files $1/**/*.gpg`
    for fn in $(find $1 -iname "*.gpg")
    do
        echo decrypt ${fn} to "${fn%.*}"
        gpg -o "${fn%.*}" -d "${fn}" 
    done
}


function remove_decrypted_files() {
    for fn in $(find $1 -iname "*.gpg")
    do
        echo removing "${fn%.*}"
        rm "${fn%.*}"
    done
}

The interface is the same: given a root directory, it decrypts all the GPG files or deletes the decrypted files.

A little bit of Bash:

• $1: refers to the first function argument, $2 refers to the second function argument and so on. This is the Bash way. When the function is called, $1 will be replaced with the actual argument, here it means the root directory.

• $(find …): is a list of files returned by the find program. In this context, it stands for all the files whose filename ends with .gpg.

  It can be achieved using ls program but it will be a lot slower ¹ and requires some configuration in MacOS ².

• ${fn%.*}: removes the last file extension of the variable $fn$, for example, foo.tar.gz.gpg becomes foo.tar.gz.

  Another approach is using $(basename $fn .gpg) to remove the .gpg extension explicitly.

• for, do, done: loops through each file.

The Bash functions have the advantage of being easily incorporated into the system, for example, call the remove_decrypted_files function automatically prior to shutting down or after login.

Footnotes

¹ why glob is slow

² how to enable globstar option in MacOS