Yi Tang

Giving Qwen 3.6 35B Vision

2026-04-25T00:00:00+01:00

Qwen 3.6 35b has been a fantastic thinking companion for me, anything that I don’t know, I am not comfortable with, or having doubts with, I would check with it. I found Qwen 3.6 + DeerFlow 2.0 is much better than the paid version of Grok, and miles better than Perplexity.

Today, I made it even better by giving it vision. Earlier I uploaded an image of my staircase and asked it to check the conditions when I plan the staircase renovation project.

This blog post highlights the key steps of how i did it.

Firstly, Qwen 3.6 has vision encoder built-in already, but it requires an additional mmproj component to make it work. Honestly I have no idea what does it mean at the moment, I just think of it as the eyes to LLM.
Download the mmproj file from the Unsloth Qwen 3.6 repo¹, add the path to –mmproj argument for llama-server command, reboot llama.cpp, that’s it.

The vision component requires additional 1-2GB of vram, so to make them fit to RTX 3090, I had to quantize the mmproj component from bf16 to q4:
```
llama-quantize mmproj-BF16.gguf mmproj-Q4_K_M.gguf Q4_K_M
llama-server Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
             --mmproj mmproj-Q4_K_M.gguf \
             ...  # rest of the llama-server arguments
```

To test it,

check the mmproj is loaded successful from the llama.cpp log,

alloc_compute_meta: graph splits = 1, nodes = 823                                                                                                                        
warmup: flash attention is enabled                                                                                                                                       
srv    load_model: loaded multimodal model, 'mmproj-BF16.gguf'

Ask Qwen 3.6 35B model to describe a small image file, using this snippet
```
curl -X POST http://192.168.1.34:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3.6-35B-A3B",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://picsum.photos/512/512"}},
        {"type": "text", "text": "Describe this image"}
      ]
    }],
    "max_tokens": 100
  }' | jq
```
This is the response I got, so it confirms it works. The image will change from time to time, so the response will be different.

The image is a scenic landscape photograph, likely taken in late autumn or winter. It features a vast mountain range in the background, rolling hills in the mid-ground covered in snow and trees, and a foreground of dry, grassy terrain. The sky is dramatic with a mix of blue and warm sunset/sunrise colors.\n\n**2. Breaking down the image into layers

if 1. success, but 2. failed, query the log file, grep vision or image, e.g. this is what I got when i misspell mmproj in llama-server at one point:

print_info: PAD token             = 248055 '<|vision_pad|>'
srv    operator(): got exception: {"error":{"code":500,"message":"image input is not supported - hint: if this is unexpected, you may need to provide the mmproj","type":"server_error"}}

The model is equipped for vision tasks, next step is to enable vision on DeerFlow 2.0, all I need is adding the support_vision to true in config, full model spec is listed below to avoid ambiguity

models:                                                                                
- name: Qwen3.6-35B                                                                    
  display_name: Qwen 3.6 35B (RTX 3090)                                                
  use: langchain_openai:ChatOpenAI                                                     
  model: Qwen3.6-35B                                                                   
  base_url: http://192.168.1.34:8000/v1                                                
  api_key: dummy_key                                                                   
  supports_thinking: true                                                              
  supports_reasoning_effort: true                                                      
  supports_vision: true                                                                
  timeout: 600    

I have to add increase the timeout to 10 mins because the vision component is a lot slower than text generation, with the default value, DeerFlow will throw errors thinking the LLM is not responding. the vision component can be optimised later to reduce the runtime, but so far so good.

Now test DeerFlow 2.0. Restart the services (make docker-stop && make docker-start), open a new chat, upload a PNG file, and ask to describe, wait for a bit, then boom!

I can also copy an image, and paste it to deerflow, which is very nice interface.

Qwen 3.6 describes an uploaded image in DeerFlow 2.0

Footnotes

¹ https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF

Multiple Working Emacs

2025-08-22T00:00:00+01:00

I work solely inside of Emacs, so when Emacs is down, I cannot do any work. Emacs itself is very reliable, but there might be some risks of downtime when upgrading Emacs or any of the 3rd party libraries that I use.

The downtime can be minimised by always having multiple Emacs versions and their 3rd party libraries available. This blog post documents how I implement it.

Installation

Firstly, install each Emacs into its separate folder, e.g. on my Debian box, I have ~/bin/emacs30.0.92/ installed 8 months ago and ~/bin/emacs30.2/ installed yesterday. This is easy to achieve by adding the prefix option when building Emacs from source, e.g.

 
./configure --with-tree-sitter  --prefix=$HOME/bin/emacs30.2

Daemon

Then have a separate systemd service for each Emacs version. Taking version 30.2 as an example, its unit file is saved as ~/.config/systemd/user/emacs30.2.service.

In that unit file, the Emacs executable is specified in full path to wherever it is installed

[Unit]
Description=Emacs text editor
Documentation=info:emacs man:emacs(1) https://gnu.org/software/emacs/
After=graphical-session.target


[Service]
Type=simple
ExecStart=%h/bin/emacs30.2/bin/emacs --fg-daemon=work --init-directory=%h/.config/emacs/emacs.d_30.2
ExecStop=%h/bin/emacs30.2/bin/emacsclient -s work --eval "(kill-emacs)"
Environment=SSH_AUTH_SOCK=%t/keyring/ssh
Restart=on-failure

[Install]
WantedBy=graphical-session.target

In the unit file, I also added the initial option init-directory so it has its own .emacs.d directory. It ensures the 3rd party packages will be installed there.

Note if there is an init.el file in that directory, Emacs will use that instead of the ancient ~/.emacs file.

GUI

Finally, to open an Emacs GUI that connects to the Emacs 30.2 daemon, run

 
~/bin/emacs30.2/bin/emacsclient -s work -c .

from the command line.

I sometimes found it is more natural to have a desktop application for GUI, so I have ~/.local/share/applications/emacsclient-30.2.desktop file, and the content is

[Desktop Entry]
Name=Emacs 30.2 (Client)
GenericName=Text Editor
Comment=Edit text
MimeType=text/english;text/plain;text/x-makefile;text/x-c++hdr;text/x-c++src;text/x-chdr;text/x-csrc;text/x-java;text/x-moc;text/x-pascal;text/x-tcl;text/x-tex;application/x-shellscript;text/x-c;text/x-c++;x-scheme-handler/org-protocol;
Exec=~/bin/emacs30.2/bin/emacsclient --create-frame -s work %F
Icon=emacs
Type=Application
Terminal=false
Categories=Development;TextEditor;
StartupNotify=true
StartupWMClass=Emacs
Keywords=emacsclient;
Actions=new-window;new-instance;

[Desktop Action new-window]
Name=New Window
Exec=~/bin/emacs30.2/bin/emacsclient --create-frame -s work %F

[Desktop Action new-instance]
Name=New Instance
Exec=~/bin/emacs30.2/bin/emacsclient --create-frame -s work %F

Not Perfect But Close

There could still be some risks of downtime due to conflicts between Emacs/package versions, or caused by updating the OS/other programs. These cases are rare, so this setup is good enough for me.

Rebate Architrave

2025-08-16T00:00:00+01:00

Had a chill day walking around the canal path in London, at 6:30 pm, I was keen to continue on the home office project.

The existing wall is not plumb, so when I put the architrave, there’s a gap. This is a typical issue, and a small gap (less than 3mm) can be filled with deco chalk. In my case, the bottom has a 10mm gap, which I have to address.

10mm gap at the bottom between the architrave and the wall

In general, there are two ways: either add a small piece to the door lining to fill the gap, or rebate the architrave to accommodate the wall protrusion. I jumped to the rebate approach as I didn’t have any additional strips of wood for the first approach (happy skip days).

A quick measurement told me the architrave needs a rebate of 45mm wide, and the depth varies: starting from 1300mm height, reaching to 10mm deep at the bottom.

The easiest way to do this in this scenario is to cut 10mm deep across the board, as it is okay to have some voids behind the architrave, and there are still 25mm for the architrave to be fixed on.

My first attempt was using a track saw: first cut was at 45mm line, from the bottom all the way to the 1300mm mark. The next cut is right next to the previous cut to increase the rebate area. Repeating this process many times to get to the whole 45mm area. The groove in the photos below is made from 3-4 passes.

With a blade kerf of size 1.8mm, I figured it requires 25 cuts to get to 45mm. My efficiency-seeking brain took over and said: There must be a better way.

So I pulled out the Dewalt router from the drawer, set the depth, and clamped the architrave down to the table. The immediate problem I faced was that it didn’t cut in a straight line: it went like 45-60 degrees for some reason, so I couldn’t cut a long groove like I had done with a track saw.

Rebate using a track saw and a router

So I turned the router 90 degree and cut small and short chunks instead. It worked well: the grooves I cut using a track saw serve as a stopping line so I won’t cut extra. It was not perfect because there were tons of dust coming out from the router, and it made so much noise.

I put my headphones on and made a few more passes. I started seeing how it can be done for the whole 1300m length. Then I saw my neighbour over the fence, asking what I was doing. Well, it turned out to be 7:30 pm already, so I had to stop and leave it for tomorrow.

In hindsight, the track saw can do a much better job because I realised only 5-10 passes would be enough. The small pieces between grooves can be knocked off rather easily using a chisel. The track saw has better dust collection, and the noise is much lower.

Another completely different approach is to remove the protrusion on the wall using a multi-tool: placing the blade on the door lining so the cuts will be flush with the door lining, and pre-cutting the 45mm line to have a neat finish.

Terminating Ethernet Cable At Height For CCTV Cameras

2025-08-06T00:00:00+01:00

On the Ground

The standard Cat 6 plug is a pain to work with: I have to untwist the 4 pairs, make them perfectly straight, lay the 8 wires side by side with no gaps, and then insert all of them into the RJ45 plug in one go.

It sounds easy, but since the wires are flexible, it is actually very hard: very often the wires move around and become misaligned or misplaced during the fitting. If that happened or any other part of it went wrong, I would have to pull out the whole lot and restart again.

I had successes before, usually after a couple of attempts, often accompanied by frustration in between. It requires me to activate the fight mode, give it 100% focus while sitting in an “Orz” position¹, so there’s quite a lot of energy poured into it.

At Height

However, even if I want to, it becomes physically impossible when it comes to fitting a plug in the air for the CCTV cameras: the ladder is a bit wobbly with uneven ground underneath it, and it is windy and raining due to a summer storm.

Since I wasn’t happy with the normal Cat 6 plug, I was keen to try new products. So when I first saw the IDC Punch Down to RJ45 Plug from Kenable ², I ordered a few. It turned out to be a smart little move (this time).

This product has a built-in RJ45 plug that is already wired up, so I can skip that difficult part. All I have to do is punch down the wires into the IDC terminal. Punching down itself is very easy; I can do it half-minded with one hand.

Another benefit is that I can split the fitting into multiple steps, and I can take mini breaks for my arms between steps. Once one or two wires are inserted into the IDC terminal, it binds the cable to the plug. The binding is strong, so it hangs in the air and swings a bit with the wind with no issues. Then I take my time for the rest of the wires. If you don’t appreciate how important it is, trust me, your arms become rather fatigued when working with your hands overhead.

The only flaw with this product is that the punch-down slots are too wide for the impact adjustable punch-down tool; I was lucky to have a tiny punch-down tool at hand to use.

Footnotes

¹ For people who don’t know what “Orz” stands for, “O” is head, “z” is legs and hips, and “r” is arms.

² This post is not affiliated with Kenable

Extend Ethernet Cable

2025-08-05T00:00:00+01:00

I had to extend the main Ethernet cable that connects the main router in the living room to the secondary router in the new office. Technically, the cable size is spot on, but I had to cut back 2-3 times because a combination of my lack of experience and the LAP data module from Screwfix is rubbish.

It seems like an unusual task given that there are only a few products available on the market. I tested two, and I am happy with the results, so I am documenting here for people who might find it useful.

Jelly Crimps

The first product I tested was from my electrician. It took me a while to find out that its name is Jelly Crimps. You can get it from TLC or Amazon.

The little connector has two long sleeves that host two wires. It has a button in the middle; press it very hard, and it will release the gel. I highly recommend using a piler unless you have super strong figures.

The process is simply: insert the wires, press with a piler to release the gel, and repeat 8 times for each wire.

Jelly Crimps in Use

It costs about £0.2 to extend one cable, so it is very cost-effective. I wasn’t sure it would work, but it does, and my electrician vouches for it.

The only problem with this product is that it is not maintenance-free. According to my electrician, I will have to put these connectors into a back box and put a front cover over it, which changes it to a much bigger job.

Inline Coupler from Kenable

So I decided to look for a better solution, and I found this Cat 6 Inline Coupler from Kenable.

It got 2 terminate blocks built-in, one for the incoming cable, and one for the outgoing. There is a diagram of the Type B protocol printed on the product, so I don’t have to look it up on my phone. All I have to do is punch down the 16 wires one by one. With a quality punch down tool it is a lot easier and quicker than I thought.

Inline Coupler In Use

The product itself is solid, much better quality than the LAP data module. I didn’t have to worry about damaging the terminal or face plate when pushing it against the wall while using a punch-down tool.

The size is on a sweet spot, about 24mm depth, just enough to tuck it into the 25mm service void. I am not sure if it is maintenance-free or not, but I am comfortable leaving it in the service void as it has an enclosing cover on it.

Kenable is the only place that sells it at a reasonable price, about £2 each, while the rest of the sellers is asking for £5 so thank you Kenable for making it affordable.

Full 500 Mbps Speed in the Office

Re-discovery the Ancient Info Documentation System in the Age of LLM

2025-04-21T00:00:00+01:00

So there are few notes that helped me to learn Info. Hopefully it can
bring more new users to the Info system.

Travel with Info
Why Info is not Popular?
dir the Index File
Setup Info in MacOS

Travel with Info

The best time to learn difficult thing is in travelling. During my last two-week’s trip to Singapore/Malaysian, I was reading about Ledger Cli causally. without putting much efforts, it clicked. It suddenly started to make sense to me.

The more I learn, the more I want to learn more. I cannot wait for the next opportunities to open Emacs and dive into Ledger’s brilliant documentation. This is me with my Emacs in Changi Airport next to the Jewe.

Emacsing next to the Rain Vortex in Changi Airport

I was able to apply the learning and came up with the project-rule to keep data hygiene (will blog next). The positive feedback energise me. The flight to London is ready for boarding but I don’t want stop exploring during the 14 hours flight without WIFI.

That’s where I re-discovered the Info documentation system. I used it to read the ledger.el library between sleep sessions 8000 feet above the ground. Reading in plain text inside of Emacs has great benefits, no distractions, fraction free in taking notes. it was a breeze.

Then I stepped into learning the Info documentation system itself, how to navigate, search text/index and all that. I was able to pick it up quickly, the concepts and shortcuts are native to me as an experienced Emacs user.

Why Info is not Popular?

I envisioned myself to use it to read all the documentation, e.g. Pandas library’s in Python. That would be ideal I told myself. However, I soon realised that Info documentation system is a niche tool: it is mostly used in GNU projects and Emacs libraries.

Why it is no popular? I was wondering myself. I decided to have a go myself. well, the journey to start is already full of hiccups. This is typical theme in learning legacy system, and could put many people off.

So there are few notes that helped me to learn Info. Hopefully it can bring more new users to the Info system.

`dir` the Index File

The first and most important thing I realised is, in the context of Info, the dir is not a directory, but a plain text file. I simply call it index file, then the rest becomes so much clearer.

The =dir=/index file is the entry point of the Info program. it has a lists of the available Info manuals with their name, Info file location, and desecration.

Setup Info in MacOS

Then there is a bug in emacs-plus: during the installation of Emacs, the dir file somehow got deleted in the cleaning process. So the manuals for the default libraries that comes with Emacs are not available. In my case, I only have few Info from the packages I installed post-installation, like orderlies, org-roam for example.

I took a slightly different approach to fix this problem: I kept the system level tools separate from the Emacs’s library, so i have two dir files.

 
# manuals of system level programs
cd /opt/homebrew/share/info
for file in * ; do install-info "$file" dir; done

# manuals of Emacs and Emacs libraries
cd /opt/homebrew/share/emacs/info
for file in * ; do install-info "$file" dir; done

Then tell Emacs the locations of those dir files as below.

 
(setq Info-directory-list
      (list "/opt/homebrew/share/info"
            "/opt/homebrew/share/info/emacs"))

Note, the convention is for each directory in the list, there is a dir file, on in Emacs, we are specifying the file using directory, and the file is happened to be called dir. I feel the naming can be improved to avoid such confusion!

After restarting Emacs, Info will show there are about 500+ manuals available, e.g. find tool, mu4e library, and Ledger3.

Lastly, an quick note install-info. As shown above, it is used to install Info manuals, taking ledger3.info as an example, to install it requires

 
install-info ledger3.info /opt/homebrew/share/info/dir

After that, the following line is added to the /opt/homebrew/share/info/dir file.

 
* Ledger3: (ledger3).           Command-Line Accounting

a bit of explanation:

*: mark the starting of the entry
Ledger3: is the node/manual name
ledger3: inside of a parenthesis is the path to the Info file without extension.
Command-Line Accounting: is the description of the manual.

Filter Ledger Transactions using Tags

2025-04-17T00:00:00+01:00

I have been testing using Ledger-Cli to track my expenses, so far I have found the tagging system useful. In my ledger journal, each transaction is associated with a project, for example, the below transaction is assigned to project “2024 Monitor Stand”

2024-12-08 Screwfix
    ; project: 2024 Monitor Stand
    Expenses:HomeImprovement:Tools            £ 4.99 
    Expenses:HomeImprovement:PPE             £ 19.98 
    Expenses:HomeImprovement:PPE             £ 14.99 
    ; :refund:
    Assets:Amex

This constraint I came up with helps avoid meaningless spending on new shiny tools. Operationally, imposing this limitation on my book provides flexible ways of querying the data.

For example, bring up the transactions that do not have projects assigned to:

 
ledger reg exp and "expr" "not has_meta('project')" \
       --format "| %(date) | %P | %(amount) | %(note) |\n"

Table 1: posts without project
Date	Payee	Amount	Note
2024/12/22	Selco	£ 30.570	; CaberFloor p5 T&G 2400x600x18mm x 2

There is only one post that I forgot to add the project tag, so pretty good.

A bit of explanation of the ledger-cli query syntax

exp: check only accounts contain ‘exp’, in the ledger’s convention, it is all expending accounts, i.e. Expense::*
expr: invoke filters using expressions
has_meta(‘project’): check if the transactions have the metadata key ‘project’
and, not: logical operators
–format: specify the output formatting

Another use case is counting the number of transactions per project. I use the number of purchased items as a proxy to gauge the project size.

 
ledger reg exp and "expr" "has_meta('project')" \
       --format "%(meta('project'))\n"  \
       | sort |  uniq -c | sort -bgr

Table 2: Number of items purchased for each project
No. Items	Project
39	2024 Loft Lights
34	2024 Loft Insulation
32	2025 Garage Conversion
8	2024 Monitor Stand
2	General

The data shows the “2024 Loft Lights” project is by far the largest . That was a simple project by itself, however, since that was my first electrical project, I had to purchase a lot of stuff, 1.5mm cables, clamps, grommets, connectors, switches, sockets etc.

Finally, I have the “refund” tag so I can flag up the items to remind of myself to check if I received the refund fully.

 
ledger reg "expr" "has_tag('refund')" \
        --format "| %(date) | %P | %(amount) | %(note) |\n"

Date	Payee	Amount	Note
2024/12/08	Screwfix	£ 14.990	Site Optimus Gel Knee Pads

So far I enjoyed the plain text accounting using ledger-cli. The format and syntax are simple, and yet I can do complicated queries.

Setup ssh-agent Systemd Service for Emacs

2025-01-26T00:00:00+00:00

Problem Statement

My personal desktop is not booting (the motherboard is probably dead) so I have been setting my server so I can work while sorting things out.

I got stuck in getting magit working in emacsclient: I thought I could run ssh-add inside of Emacs that would allow magic to access my git repos using ssh, but apparently, it is not the case.

After some digging, I learnt that the problem I have to solve is to run one ssh-agent in the background and then make the Emacs/Magit or any programs hook onto it. Then once I run ssh-add and type the passphrase for the first time, either inside of Emacs or in a bash terminal, everything would work.

Implementation

Drop the following unit file below to ~/.config/systemd/user/ssh-agent.service.

[Unit]
Description=SSH key agent

[Service]
Type=simple
Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket
ExecStart=/usr/bin/ssh-agent -D -a $SSH_AUTH_SOCK

[Install]
WantedBy=default.target

The important things are

The environment variable SSH_AUTH_SOCK is specified. It can be anywhere as long as this environment variable in other programs points to the same location.
ssh-agent is invoked with the -a option to provide an address specified in the above step.

The $t is a specifier¹ in systemd, it is equivalent to $XDG_RUNTIME_DIR variable in Debian. It points to the runtime temporary directory which apparently is safer² than the /tmp directory. The runtime directory was cleaned up after stopping the ssh-agent so it is non-persistent.

To start the ssh-agent service:

 
systemctl enable --user ssh-agent
systemctl start --user ssh-agent

After that, update the unit file of Emacs to include this line (follow up my blog post Managing Emacs Server as Systemd Service for the full setup).

Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket

To make it work for bash shell and all other programs calling from a bash terminal, add this line to ~/.bashrc.

 
export SSH_AUTH_SOCK="$XDG_RUNTIME_DIR/ssh-agent.socket"

Alternatives

There are programs developed to solve this specific problem (see Debian wiki). While using such a program seems like a simpler alternative (e.g. keychain), I prefer to use systemd as the unified approach for managing background services. I have been using it for emacsclient, and I’m adding ssh-agent to it.

What is your preference? How do you solve this problem?

Footnotes

¹ All the specifiers are listed here.

² I am not a security expert but the StackExchange comments seem to make sense.

Retiring Raspberry Pi 4 as Home Server and NAS

2025-01-20T00:00:00+00:00

Good Start for Self-Hosting
Lack of NAS Capacity
Looking for a Successor
Unexpected
Setting up z170a
Power Consumption

Good Start for Self-Hosting

The little Raspberry Pi 4 (RP4) served me well in the last two years. I used it to host NextCloud/Syncthing for syncing files between devices, scraping financial data from Yahoo Finance and TimeMachine for MacOS backup.

The latest addition to the service stack is paperless-ngx. It allows my Canon printer/scanner to send digital copies of documents directly to the RP4 or Gmail.

The RP4 handles all the demands without showing any signs of struggle. It costs as little as 6kW per hour while the Xbox One S draws 11kW while sleeping. Thanks to the energy crisis in the UK, I started to appreciate the energy efficiency of RP4. The ARM chips in it really impressed me.

Lack of NAS Capacity

A 3TB portal hard drive (WD My Passport) was attached to the PR4 to store media data. The USB 3.0 connector is surprisingly stable and fast. With both ends connected by ethernet cables, the file transfer speed can reach up to 100 MB/s. When my MacBook Pro uses Wi-Fi, the speed drops to about 40-50 MB/s but it is still great because of the convenience.

Later I started using it as a NAS to store the Final Cut Pro library. The 4k home gym videos I shot using iPhone 12 Pro are numerous ¹! The hard drive keeps getting filled up.

I can get another portal hard drive, but then it will get filled up again, say in less than a month? So it occurred to me that I need a proper home server with full NAS capacity.

Looking for a Successor

I did a bit of research but I am not able to find a good product. I suspect the reason is the NAS build is a niche area while the PC industry is gaming-centric, focusing on getting faster, bigger, and fancier hardware with unnecessary RGB lights, that is where the profits are I presume.

I came across some innovative products on AliExpress from China, such as the TopTon N5105 board. It is more powerful, consumes slightly more electricity, and it has 6 SATA cables! It would be a perfect successor for my PR4.

But I am not comfortable ordering electronic stuff from AliExpress, returning it or sending it back for repair would be a nightmare.

PS: The company is growing fast, it continued to innovate, and the product lines extended to Intel N100 with an additional NVME drive and a USB-C. Their website and marketing materials look notched up quite a bit. I kind of regret not taking the risk back then.

Unexpected

The other day, I was re-organising (again) my home office, so had to move a bookshelf. I started moving it without taking everything off, then a motherboard fell off. It was the z170a with an i5-6600k and a heat sink attached to it. The motherboard was in my first desktop that I purchased 10 years ago when I started participating in Kaggle competitions in 2014.

After a quick inspection, I saw some pins were bent. I felt ashamed and sorry for the motherboard that I had not taken care of it. So I made a promise: if it survived the fall, I would use it for my NAS.

Well, it did so I found my NAS.

Setting up z170a

While putting it up, one SATA port was snapped and came up, but the rest is still fine. Apart from that, everything else went smoothly. The Debian 12 became much easier to install with the isohybrid technology and the non-free firmware is now part of the installation image itself.

The server setup scripts and configuration are saved in a selfhosted-services git repository so restoring the services took little efforts.

I had one little trick: I assigned the IP address of RP4 to the new z170a server so that on the client side I didn’t have to change anything. This was achieved rather easily: few clicks in the ASUS router web UI and then a reboot.

While setting it up, I noticed the z170a system is much more responsive, thanks to the 3.5 GHz i5-6600k CPU and a much faster SSD over the SD card. I was able to run multiple processes at the same time.

The longest part is copying files from the 3TB portal hard drive to the z170a’s internal HDD, which took about 20 hours.

It has great extensibilities: there are 3 free SATA for HHD and two PCIe slots.

Power Consumption

The only downside is that it consumes a lot more electricity. When testing in barebone, it drew only 10W. After putting everything together with additional HDDs, fans, and ethernet cable, the power metre jumped to 45W. I removed hard drives one by one to see where the bottleneck is.

No HDD, 27W
IronWolf alone, 32W, 5W increases.
IronWolf + Seagate, 37W, another 5W increase.
IronWolf + Seagate + Toshiba, 45W, 8W increase.

So I kept only IronWolf which is a 3TB NAS grade HDD.

I also tried tweaking the BIOS and Linux kernel to get to C-states but I felt it was over-engineering so I am happily settled down with 27W.

Footnotes

¹ I record weightlifting to correct and improve my techniques.

Use Ledger-Cli to Track DIY Project Expenses

2025-01-14T00:00:00+00:00

Personal Technical Challenge
Baby Steps
Why? - Effort Estimation

Personal Technical Challenge

I used ledger-cli¹ before and it was a painful experience. The problem was not rooted in the tool but in how I intended to use it: I wanted to track all my expenses, from buying a cup of coffee to booking a holiday package. When I started this journey, there was a massive jump from knowing little to nothing about personal finance to doing double-entry accounting in plain text.

Though I gave up, it introduced me to the idea of owning my bank transaction data in text files on my personal computer. So over the years, I manually curated about 8 years of historical transaction data.

If you haven’t done so, I strongly recommend you go to your banks’ website and download the transaction data manually, going as far back as you can. You will notice that the banks only give access to 3-5 years of data². It’s a shame that banks use outdated technologies but it is better than having nothing.

Since I had the data, I did some analysis and charts in Python/R. But I kept wondering what ledger-cli can offer. I occasionally saw blog posts on ledger-cli in the Emacs communities, so there must be something out there.

It also has become a personal challenge. I turned not to give up but put it aside to tackle it again after I got older.

Baby Steps

Hopefully, I had become smarter as well. This time, to ensure I can successfully adopt the tool, I am going to reduce the scope to limit to only tracking DIY project expenses.

I love DIY and I wish I had more days for DIY projects. It is usually labour-intensive and I feel hyped and extremely confident after a couple of DIY. Pairing it with learning ledger-cli, a cognitive-intensive activity, would make them a nice bundle³.

Though the usage is simple, the question it can answer is important. I want to know, during or after the DIY project, how much it exactly costs. I could use a much simpler tool, like spreadsheets or a pen/notebook, but I want it to be a stepping stone to acquire ledger-cli properly in the future.

Why? - Effort Estimation

I need an accurate answer to the actual costs so that I can use the data to train myself in cost estimation. This is an very important skill to have as a homeowner, it would put me in a much better position in negotiation with the tradesman. A lot of the people in the UK complained that they or their relatives got ripped off by tradesman.⁴

In general, house repairs and improvements are getting much more expensive every year, due to the shortage of labourers, inflation and Brexit etc. To give an example using my last two quotes, adding an electrical socket costs £240 and replacing a small section of water pipes costs £500.

I have a good habit of using org-mode to track time, my goal to add ledger-cil to my system to track the expenses. After that, I would know if it is really worth doing the DIY or finding a proper tradesman. The total cost itself is not the only metric that matters, but n very essential one to have.

Footnotes

¹ https://ledger-cli.org/

² Why don’t banks give access to all your transaction activity?

³ I might pick it up from Atomic Habit

⁴ How many of you have been ripped off by builders / tradesmen? (or know someone closely that has)

Finding Highly Correlated Features

2024-10-20T00:00:00+01:00

Motivation
Implementation
Parameterisation

Motivation

From a modelling perspective, it is not a big problem to have highly correlated features in the dataset. We have regularised Lasso/Ridge regression that are designed to deal with this kind of dataset. The ensemble trees are robust enough to be almost immune from this. Of course all model requires proper hyperparameter tuning with proper cross validation.

The problem raises in understanding the feature contributions: if there are 5 features that are highly correlated, and their individual contribute could be tiny, but their true contribution should be aggregated by adding the contribution together and considered them as a group, e.g. adding their coefficients in Ridge, and adding feature importance in LightGBM.

If their aggregated feature importance turns out to be indeed little, I can remove them from the model to have a simpler model. A mistake I used to make is removing the correlated features based on their individual feature importance, it leads to less performant models.

A better and cleaner approach is to the clean up correlated features to begin with, then I won’t need to do the feature importance aggregation, and it would speed up the model development cycle: there are less features to look at, to train the model, to verify the data qualities etc. When the model goes live in production, it translates to less data to source and maintenance.

Implementation

So I need to enrich my tool set to identify highly correlated features. I couldn’t find an existing library that does that, so I implemented it myself.

The key steps are:

Based on the correlation matrix, create a correlation long table. Each row stands for the correlation between feature $X_1$ and feature $X_2$. Assuming there are three features in the dataset, the table looks like this.

Row X1 X2 Corr

1 A B 0.99

2 A C 0.80

3 B C 0.95
Remove rows if the correlation is less than the threshold $T$. It significantly reduces the input to Step 3.

If the threshold is 0.9, then the Row 2 will be removed.
Treat the correlation table as a directed graph,
1. Let $E$ be the unexplored nodes, filled with all the features $X_1$ in the start, $R$ is the result.
2. For each node in $E$,
  1. Continue to travel the graph in depth-first fashion until there is no connections left, and add the connected node to the result $R$ at each visit.
  2. Remove the connected nodes in $R$ from the remaining nodes to explore in $E$.

Row	X1	X2	Corr
1	A	B	0.99
2	A	C	0.80
3	B	C	0.95

The vanilla Python code corresponding to Step 3 is listed below. The ds object is a pandas.DataFrame, multi-indexed by $X_1$ and $X_2$, so ds.loc['A'].index gives all the connected features from $A$ whose correlation with $A$ is large than the provided threshold.

 
def find_neighbors(ds: pd.DataFrame, root: str, res: set):
    """recursively find the nodes connected with root in graph ds.
    """
    res.add(root)
    if root in ds.index:
        ns = ds.loc[root].index.tolist()
        for n in ns:
            find_neiboughr(ds, n, res)
    else:
        return []

def find_correlated_groups(ds: pd.DataFrame):
    """
    The ds object is a pandas.DataFrame, multi-indexed by X1 and X2.
    """
    res = defaultdict(set)

    # contiune til all nodes are visited.
    cols = ds.index.get_level_values(0).unique().tolist()
    while len(cols) != 0:

        # always start from the root as ds is directed graph.
        col = cols[0]
        find_neighbors(ds, col, res[col])

        # remove connected nodes from the remaining.
        for x in res[col]:
            if x in cols:
                cols.remove(x)
    return res

The result is a collection of mutually exclusive groups. Each group contains a set of highly correlated features, for example

Group A: {A, B, C} Group D: {D, K, Z}

The next step is to decide which feature to keep and remove the rest within each group. The deciding factors can be data availability (e.g. choose the one feature with less missingness), costs in data sourcing (e.g. free to download from the internet) or familiarity (e.g. the feature is well understood by people) etc.

Parameterisation

There are two hyperparameters:

The correlation type: It can be Pearson for numerical data and Spearman for ordinal/categorical data. For a large dataset, it would take some time to calculate the correlation matrix.
The correlation threshold $T$: The higher the threshold, the less number of features to remove, so it is less effective. However, if the threshold is set too low, it leads to a high false positive rate, e.g. two features can be correlated, but they can still complement each other in the model.

I would test a range of values from 0.9 to 1, and review the results. Below graph shows the number of features to remove with varying thresholds.

When $T=0.9$, there are about 95 groups, and in total 153 features to remove.
When $T=1$, there are 29 groups, and in total 37 features to remove.

Proper end-to-end test runs are required to identify the best hyperparameters. As a quick rule of thumb, those 37 duplicated features identified with $T=1$ can be dropped without further testing.

The group sizes with varying thresholds $T$ provide an interesting insight of the data. The 75% percentile of the group sizes is plotted, which suggests that apart from the 33 duplicated features, there are a large number of paired features (i.e. group size is 2) whose correlation is large, more than 92%.

Less Excel, More R/Python in Emacs

2024-04-10T00:00:00+01:00

Excel Is Great
But
Emacs Has More To Offer

Excel Is Great

Regardless of how powerful and convenient the R/Python data ecosystem becomes, there is still value in looking at the data in Excel, especially when exploring the data together with less technical people.

Thanks to its trivial interface Excel is widely used in data analysis: hoover the mouse to select columns, apply filters then calculate some statistics. Most of the time that is all it takes to get the answers the clients are seeking.

I recently realised that having transparency and working with the tools that clients use plays a crucial role in strengthening the trust and delivering the impacts to the business. Sometimes I think I should do more in Excel.

But

The problem with Excel is reproducibility - I’m not able to codify the clickings done in Excel and integrate them into the automated data pipeline. It is rather foolish to have quality control procedures, including code reviews, automated testing, CI etc in the system but in the very end drop all those gatekeepers and go for error-prone manuals.

Plus it is way more efficient to have everything done in one place to have a smooth process with no fractions. It is a key factor in enabling quick turnaround.

So I had the motive to limit the usage of Excel to deliver data to the business and pair data analysis. Again I have been looking into how much it can be done without leaving Emacs.

Emacs Has More To Offer

I was pleased to discover the ess-view-data package and its Python counterpart python-view-data. They interact with an active R/Python session in Emacs and print out data.frame objects in plain text, a.k.a. view data. What’s more, it can process the data before viewing, for example, subset the data row/column-wise, summarise the dataset etc.

The package keeps a record of the data processing pipeline so in the end I would have a copy of the R/Python code that generates the output. I can then effortlessly transfer the code to a script to ensure reproducibility in the future.

Another benefit derives from having a plain text buffer for the data. It is handy in exploring large datasets with an excessive number of columns. For example, the dataset I work on daily basis has about 300 columns. It contains different flavours of the financials, the raw values, imputed, ranked, smoothed etc.

It’s not possible to remember all the column names even after more time was spent in giving meaningful names or ensuring the correct columns are referred to. Having a persistent plain text buffer that I can search for makes finding the right column names a lot easier. It also helps to check what’s in and not in the data.

That’s my first impression of Shuguang Sun’s packages, it looks promising.

Blog in Emacs - Use Jekyll's Draft Mode

2024-02-12T00:00:00+00:00

Why?

I wasn’t aware of Jekyll’s draft mode. My workaround was manually changing the published field in the front matter to true when the post is ready to publish. It works fine. However, with naive support from Jekyll, there are more benefits to using the draft mode.

To start with, I like the drafts saved in the _drafts folder, not mixed with other published posts in the _posts folder. It is way more cleaner and easy to manage. With a glimpse of my eyes, I can see what are the posts that I am drafting.

It also gives a piece of mind: only posts under the _posts folder are exported and shown in my blog. It ensures I don’t accidentally publish a post in draft.

Once there are files in _drafts folder, adding --drafts argument to the jekyll serve command is all I need to be able to see the drafts locally.

Of course, I also need to write a bit of Lisp code to integrate the draft mode into my blogging workflow. This is the remaining of this post is about.

Implementation

For a blog post, I have the source file in org-mode and its exported file in Markdown. Now there is a new location dimension: they can be either in the _drafts or _posts folder.

mode	source file (org mode)	exported md in Jekyll
draft	org/_drafts/on_image.org	jekyll/_drafts/on_image.md
publish	org/_posts/2027_02_08_on_image.org	jekyll/_posts/2027_02_28_on_image.md

In terms of content, the published post and its final draft, and their exported counterparts are the same, only in different locations. Their content can be different to have some flexibility, e.g. published post has higher resolution of screenshots. This feature is possible to implement in the future. For now, I follow the simple “same but in different places” rule.

The new process looks like this: When I publish a post, it moves the org file from _draft to _posts folder, adds a date to the filename (which I have already), and then triggers the exporting process. To avoid duplication, it removes the original org file and its exported draft in Markdown.

To achieve that, the main missing piece from my current Emacs configuration is the yt/jekyll-find-export function (see below). For a post in _drafts or _posts, it finds the full path of the corresponding exported markdown file. I can then delete it or start the exporting process.

 
(defun yt/jekyll-is-draft-p ()
  "if the file is inside of the draft directory, it is a draft."
  (let ((draft-dir  (file-truename jekyll-source-drafts-dir))
        (filepath (file-truename (buffer-file-name))))
    (string-prefix-p draft-dir filepath)))


(defun yt/jekyll-find-export ()
  "find the full path to the exported file of the current post."
  (let* ((src-file (file-name-nondirectory (buffer-file-name)))
         (dest-file (file-name-with-extension src-file ".md")))
    (if (yt/jekyll-is-draft-p)
        (file-name-concat jekyll-site-draft-dir dest-file)
      (file-name-concat jekyll-site-post-dir dest-file))))

Blog in Emacs - Work with Images

2024-02-03T00:00:00+00:00

I do my best to keep my blog simple, I would not use images/videos unless I can’t demonstrate well enough in plain text, for example, to demonstrate a mobile app using a screenshot (Learn in Emacs - Building Up Vocabulary) or how to represent stock price charts for neutral network (Speed Up Sparse Boolean Data).

Even when I do, I keep the usage to the bare minimum: all I do is insert the image, make it centralised, and put a caption on top of it.

The Org-mode supports images well, with a few additional HTML attributes for each inserted image, I can fine-control the images’ position, alignment, size etc.

However, I can’t get the benefits because I migrated my blog posts from HTML to the Markdown format for its simplicity. Plus Jekyll comes with its little quirks when it comes to Markdown images. So I have to write something for myself.

I managed to achieve a satisfactory workflow for my simple usage of images in Emacs. It works well for the Jekyll site. Here’s the code and explanation.

org-download : is the package that I use to create the images for blogging from various sources.

I can drag images from external applications to Emacs, including browsers, Preview, or iPhoto. The images will be saved in the /project/assets/org-download folder per my matrix/project setup.

For the application that I can’t drag the images, I take a screenshot inside Emacs by calling the org-download-screenshot function.
yt/jekyll-copy-from-org-downkload: is a little helper function that transfers the files under the org-download folder to the /assets folder in a Jekyll site.

It lists the files in the source org-download folder and provides them as a selection list. It comes with auto-completion and fuzzy matches to help me choose the file.

It also strips out the special characters in the filename otherwise the URL will be broken in Jekyll.
yt/jekyll-insert-image: lists the files in the /assets folder so I can choose easily which image to use.

It brings up the Liquid template for image so I don’t have to remember its syntax. It ensures the file path is in the correct format (starts with /assets/), I just fill in the caption and size after selecting the file.

An extract of the code is listed below for demonstration propose. Future updates will be reflected in my .emacs.d git repo.

 

(defun yt/jekyll-insert-image (src caption)
  (interactive (list (read-file-name "images to include: " jekyll-assets-dir)
                     (read-string "Caption: ")))
  (insert (format jekyll-insert-image-liquid-template (file-name-nondirectory src) caption)))

(defun yt/jekyll-copy-org-download-to-assets (file)
  "copy file from project org-download folder to the blog assets folder.
it ensures there's no underscore(_) in the file name.
"
  (interactive (list (read-file-name "file to copy: " org-download-image-dir)))
  (let* ((ext (file-name-extension file ))
        (base (file-name-base file))
        (dest-base (jekyll-make-slug base))
        (dest-file (expand-file-name (file-name-with-extension dest-base ext) jekyll-site-assets-dir)))
    (copy-file file dest-file)
    dest-file))

Learn in Emacs - Building Up Vocabulary

2024-01-28T00:00:00+00:00

WHY?
Workflow for Building Vocabulary
Revision on Mobile Devices
Org-mode Based Simple Study Strategies
Emacs Lisp Implementation

WHY?

Research shows having effective and rapid communication can boost creativity and spark joy¹. I believe in it from my personal experience in conversing, reading a book in my native language or trying to understand a large codebase.

I wasn’t enable to achieve similar results when it came to using English. In the past I have been trying to improve my English language skills to boost my productivity in reading books and to make it more enjoyable. The approach was practising more in reading and writing. However, I started questioning the effectiveness. This year I decided to take one step back to focus on the basics and improve my vocabulary.

I want to take the “slip-box” method² which proved to be effective for learning Emacs Lisp language. It is a bottom-up approach, so I would have one note for each word with the explanation in it, links to other similar words, or words I got confused with.

One advantage is that I can also leverage my existing setup.

Workflow for Building Vocabulary

When come across a new word that I’m not sure about its meaning, I will

move the cursor to the word,
press F1 d to look into the dictionary, the result will shown in the osx-dictionary buffer,
read its meaning and try to understand it,
press r to listen the pronunciation and read after it. I usually repeat it a couple of times to deepen the memory,
press a to create an atomic note. it has the dictionary meaning in it for future reference,
edit the notes to add my understanding and copy the sentence/paragraph that contains the new word.
press C-c C-c to save it to my vocabulary database, which is just a folder with flat org-mode files.

There’s quite a lot of automation so I can focus on understanding it (Step 3) and write a good note (Step 6) in my own words.

This workflow depends on two Emacs packages:

osx-dictionary: it interfaces with macOS’s dictionary app. It displays the meaning and says the pronunciation.

The package is well written and easy to work with; I managed to extend it to add Steps 5-7 with little effort.

It has limitations: it works only in macOS and it only outputs one dictionary. Adding the meaning in Chinese requires a few more manual steps: 1) press o to open the Dictionary.app, 2) go to the Chinese dictionary tab and copy the meaning, and 3) paste it to the note in Emacs.

I personally find the Oxford dictionary macOS uses is not easy to follow. From time to time I have to visit https://dictionary.cambridge.org/ to find the explanation that I could understand. In transforming my old vocabulary notes to the new format, I found the explanation from vocabulary.com is the best. I might have to resurrect my voca-builder³ package.
org-roam: it interfaces with org-mode for creating atomic notes. It avoids duplication: if there’s a note for the word that exists already, it opens the note, so I can have a look and enrich it.

I can link notes/words in my vocabulary database which is very useful because for me learning by comparing is super effective.

The org-mode provides a lot of functionalities that might be useful to facility learning in the future.

Revision on Mobile Devices

Once I have a fleet of notes, the next step is to revise them on a regularly. The routine I’m trying to get myself into is rereading the notes I created for the last few days while waiting for the tube/bus, I call it a revision break.

So far I have an Emacs lisp program⁴ that filters all my notes by time so I have last_24_hours.org, last_3_days.org and last_7_days.org. These files are synced with iCloud so they are available to review on my iPad and iPhone using the beorg App.

Reading my vocabulary notes on iPhone

Org-mode Based Simple Study Strategies

For the dedicated study sessions, I need a few strategies to shortlist the notes. I think They will be based on the metadata of the note. With org-mode’s API, it should be easy to implement.

I haven’t done it yet, but the idea is to score the notes from 0 to 5, 5 means the most important notes so I would study them first, 0 means not important notes so will be at the bottom.

There can be multiple scores, for example, one for pronunciation. the word that I got the pronunciation completely wrong would get a 5, and the word would get a 3 if sometimes I got it wrong, and sometimes I got it right.

Another score is how many times I looked into the word. There are words that I just keep forgetting about it, or keep confusing with another similar word. So the property of ‘visited_at’ gets a timestamp appended at the time of visiting and the score is calculated by the number of timestamps.

Emacs Lisp Implementation

Adding an action to the headline in osx-dictionary’s buffer.

 
(require 'osx-dictionary)
(setq osx-dictionary-mode-header-line
      (append '((:propertize "a" face mode-line-buffer-id)
                ": Add to vocabulary"
                "    ")
              osx-dictionary-mode-header-line))

Adding yt/add-to-vocabulary to key a in osx-dictionary buffer. It creates an note using org-roam.

 
(defvar vocabulary-repo-dir "~/matrix/learning/meta-leanring/vocabulary"
  "where to save the vocabulary notes.")
(defvar yt/voca--roam-template
  '(("d" "default" plain "%?" :target
     (file+head "%<%Y%m%d%H%M%S>-${slug}.org" "#+title: ${title}

%?

#+begin_example
%(yt/osx-dict--get-meaning)
#+end_example

")
     :unnarrowed t))
  "roam template for vocabulary notes")


(defun yt/add-to-vocabulary ()
  "add a new vocabulary note for ther highlihted region or word at
point."
  (interactive)
  (let* ((org-roam-directory (expand-file-name "notes" vocabulary-repo-dir))
         (org-roam-db-location (expand-file-name "org-roam.db" org-roam-directory ))
         (org-roam-capture-templates yt/voca--roam-template))
    (org-roam-node-find nil (yt/osx-dict--get-word-and-pronounce))))

(defun yt/osx-dict--get-word-and-pronounce ()
  "extract the word and its pronunciation from the *osx-dictionary* buffer"
  (with-current-buffer "*osx-dictionary*"
    (goto-char (point-min))
    (search-forward "|" nil nil 2)
    (buffer-substring-no-properties (point-min) (point))))

(defun yt/osx-dict--get-meaning ()
  "wrap the *osx-dictionary* buffer cnotent as a string"
  (with-current-buffer "*osx-dictionary*"
    (buffer-substring-no-properties (point-min) (point-max))))

(define-key osx-dictionary-mode-map "a" 'yt/add-to-vocabulary)

Footnotes

¹ reference is lost; it is somewhere in the book “The Second Mountain”, the chapter on religions.

² from the book “How to Take Smart Notes”. I plan to reread this book in early 2024.

³ My first Emacs package in 2015, https://github.com/yitang/voca-builder

⁴ next blog post is on my lisp programs

Atomic Habit in Emacs - Keep Git Repos Clean

2024-01-14T00:00:00+00:00

Why?
Emacs Lisp Helper
Practise

Why?

I am having a hard time keeping my git repositories clean: there are just too many repositories, I counted 31 in total, and I have 5 computers where I work on them.

The consequence is that sometimes I get surprised at seeing a lot of seemingly useful changes that are not committed to the git repo. I had to stop whatever I was doing to just think about what to do with those changes. It breaks the flow!

There are other occasions where I thought I fixed some bugs, but I don’t have the patches on my laptop. It turned out I didn’t check in to the cloud, so I have to log back to the right server to run a couple of git commands, or if I don’t have access to the servers, I have to fix the bugs from scratch again. It is inefficient!

It can happen a lot in active projects where I work on multiple systems and multiple git repos or when I travel. I plan to revisit my filesystem (which is inspired by Stephen Wolfram ¹) and tech setup to reduce the number of repos by merging them and keeping only 1 laptop, 1 workstation and 1 server. This is something for summer, it can reduce the severity of the problem but can not eliminate it.

At the moment, I just have to become more disciplined in managing files, e.g. to have an atomic habit of checking my git repo regularly, or at least do it once at the end of the day, or as part of the shutdown ritual after finishing a task².

Emacs Lisp Helper

The 3rd Law of Behavior Change is make it easy.

James Clear, Atomic Habit

To facilitate the forming of this habit, I implemented a utility function in Lisp to list the dirty git repo, and provide a clickable link to the magit-status buffer of the git repo. With one click on the hyperlink, I can start to run git commands via the mighty magit package. I bind this action to keystroke F9-G.

 
(defun yt/git--find-unclean-repo (root-dir)
  ""
  ;; (interactive)
  (setq out nil)
  (dolist (dir (directory-files-recursively root-dir "\\.git$" t))
    (message "checking repo %s" dir)
    (let* ((git-dir (file-name-parent-directory dir))
           (default-directory git-dir))
      (unless (string= "" (shell-command-to-string "git status --porcelain"))
        (push git-dir out))))
  out)


(defun yt/dirty-git-repos (&optional root-dir)
  "list the dirty git repos, provides a clickable link to their
magit-status buffer."
  (interactive (list (read-directory-name "Where's the root directory?" )))

  (let ((buffer (get-buffer-create "*test-git-clean*"))
        (git-repos (yt/git--find-unclean-repo root-dir)))
    (with-current-buffer  buffer
      (unless (eq major-mode 'org-mode)
        (org-mode))
      (goto-char (point-min))
      (insert (format "Number of dirty git repos: %s " (length git-repos)))
      (dolist (git-repo git-repos)
        (insert (format "\n[[elisp:(magit-status \"%s\")][%s]]" git-repo git-repo))))
    ))

The workhorse is the git status --porcelain command: If the git repo is clean, it returns nothing, otherwise, it outputs the file names whose changes are not checked in, e.g. the first file is modified (M), and the second file is not untracked (??).

 M config/Dev-R.el
?? snippets/org-mode/metric

The rest of the code is for parsing the outputs and turning them into a user-friendly format in Org-mode. What’s interesting is that The org-mode provides a kind of hyperlink that evaluates Lisp expressions, using the example below,

 
[elisp:(magit-status "/foo")]["Git Status of Repo /foo"]

The description of the hyperlink is “Git Status of Repo /foo” , after I click it, it runs the expression (magit-status "/foo") which shows the git status of /foo repo in a dedicated buffer.

Before executing it will ask for a confirmation. It can be a bit annoying and inconvenienced at first which naturally leads to the temptation of removing this behaviour by setting org-link-elisp-confirm-function to nil. I discourage you from doing so in case someone embeds funny codes, (for example rm -rf ~/) in a hyperlink, so make sure to check that variable’s documentation before changing it³!

Practise

It was fun to write the lisp functions. I learnt how to use the optional function argument and interactive so that the function can be used both interactively and pragmatically. I’m very much wanting to spend more time in coding, to enhance it with some ideas I got from reading Xu Chunyang’s osx-dictionary package⁴.

However, the effectiveness of those functions has little to do with the extra features I had in mind but really depends on how I use them. Solving the problems requires deliberate practise and changing my behaviours so that cleaning git repos becomes a habit of mine, which is always the hardest part.

One key indicator for this habit⁵ can be the number of check-ins and see if there’s a substantial increase from today.

Footnotes

¹ see Stephen Wolfram’s blog posts

² Cal Newport, Deep Work, Page 151

³ https://orgmode.org/manual/Code-Evaluation-Security.html

⁴ https://github.com/xuchunyang/osx-dictionary.el

⁵ inspired by Andrew Grove’s book High Output Management

GPG in Emacs - Functions to Decrypt and Delete All

2024-01-06T00:00:00+00:00

Motivation
Emacs Lisp Implementation
Bash Implementation

Motivation

Continuing from my last post, the EPA provides a seamless interface when working with GPG files in Emacs. But there are situations where I have to work with GPG files using other programs (mostly Python) which EPA cannot help.

For those cases, I have to decrypt the GPG files first before using them (for example, calling pandas.read_csv).

Obviously, there’s no point in encrypting a file if there is a decrypted version next to it. So I also need a function to delete all the decrypted files.

Emacs Lisp Implementation

Of course, I run Python inside of Emacs, I wrote the Lisp functions to decrypt GPG files and delete all the decrypted files.

 
(defun yt/gpg--decrypt-recursively (root-dir)
  "It decrypts all the files ends .gpg under the root-dir. The decrypted files have the same filename but without the .gpg extension.

It stops if the decryption fails. 
"
  (interactive)
  (dolist (file (directory-files-recursively root-dir "\\.gpg"))
    ;; the 2nd argument for epa-decrypt-file can only be the base filename without the directory.
    (let ((default-directory (file-name-directory file)))
      (epa-decrypt-file file (file-name-base file))
    ))
  )

(defun yt/gpg--delete-decrypted-files (root-dir)
  "It deletes the decrypted files under the root-dir directory.

e.g. if there's a file foo.tar.gz.gpg, it attempts to remove the foo.tar.gz file.
"
  (interactive)
  (dolist (file (directory-files-recursively root-dir "\\.gpg"))
    (delete-file (file-name-sans-extension file))
    )
  )

A bit of explanation:

directory-files-recursively: searches for files with a pattern. Here, it returns all the files ending with .gpg under the given root-dir,
dolist: loops over the GPG files to process them one by one,
epa-decrypt-file: decrypts a GPG file into a new file.
delete-file: deletes a given filename.

It seems the epa-decrypt-file function does not like the new filename with the directory in its path, so I have to set the default directory (working directory) and use the base filename after removing the directory as a workaround.

Bash Implementation

It would be useful to have those functionalities outside of the Emacs, so I implemented their counterpart in Bash.

 
function decrypt_recursively() {
    # PS: this function is equivalent to `gpg --decrypt-files $1/**/*.gpg`
    for fn in $(find $1 -iname "*.gpg")
    do
        echo decrypt ${fn} to "${fn%.*}"
        gpg -o "${fn%.*}" -d "${fn}" 
    done
}


function remove_decrypted_files() {
    for fn in $(find $1 -iname "*.gpg")
    do
        echo removing "${fn%.*}"
        rm "${fn%.*}"
    done
}

The interface is the same: given a root directory, it decrypts all the GPG files or deletes the decrypted files.

A little bit of Bash:

$1: refers to the first function argument, $2 refers to the second function argument and so on. This is the Bash way. When the function is called, $1 will be replaced with the actual argument, here it means the root directory.
$(find …): is a list of files returned by the find program. In this context, it stands for all the files whose filename ends with .gpg.

It can be achieved using ls program but it will be a lot slower ¹ and requires some configuration in MacOS ².
${fn%.*}: removes the last file extension of the variable $fn$, for example, foo.tar.gz.gpg becomes foo.tar.gz.

Another approach is using $(basename $fn .gpg) to remove the .gpg extension explicitly.
for, do, done: loops through each file.

The Bash functions have the advantage of being easily incorporated into the system, for example, call the remove_decrypted_files function automatically prior to shutting down or after login.

Footnotes

¹ why glob is slow

² how to enable globstar option in MacOS

GPG in Emacs - First Step Towards Data Security

2023-12-28T00:00:00+00:00

WHY?
GNU Privacy Guard (GPG)
EPA - Emacs Interface to GPG
Org-Agenda and Dired
Lisp to Close all GPG Files

WHY?

I have growing concerns about data security. It is not that I have something to hide, it’s that I don’t like how my data is being harvested in general by the big corporations for their own benefits, which is mostly trying to sell me stuff that I don’t need or I purchased already. Seeing the advertisements specifically targeting me motivates me to do something.

Setting my personal cloud seems a bit too extreme, and I don’t have the time for it anyway. So I did a little “off-the-grid” experiment in which I exclusively used an offline Debian laptop for data sensitivity work (password management, personal finance, diary etc). It is absolutely secure for sure, but the problem is accessibility: I can only work when I have access to the physical hardware.

It becomes infeasible when I travel, and it gives me some headaches to maintain one more system. Also, the laptop’s screen is only 720p, I can literally see the pixels when I write; it feels criminal to not use the MBP’s Retina display. Lastly, It cannot be off the grid completely; at one point, I have to back it up to the cloud.

So I spent some time researching and learning. I just need a data protection layer so that I don’t have to worry about leaking private data accidentally by myself, or the cloud storage provider getting hacked.

The benefits include not only having peace of mind but also encouraging myself to work on those types of projects with greater convenience.

GNU Privacy Guard (GPG)

is the tool I settled with. It is a 24 years old software that enables encrypting/decrypting files, emails or online communication in general. It is part of the GNU project which weighs a lot to me.

There are two methods in GPG:

Symmetric method: The same password is used to both encrypt and decrypt the file, thus the symmetric in its name.
Asymmetric method: It requires a public key to encrypt, and a separate private key to decrypt.

There seems no clear winner in which method is better¹. I choose the asymmetric method simply for its ease of use. The symmetric method requires typing the passwords twice whenever I save/encrypt the file which seems too much.

The GPG command line interface is simple. Take the below snippet as an example,

 
gpg -r "Bob" -e foo.org
gpg -o foo2.org -d foo.org.gpg

The first line encrypts the foo.org file using the public key identified as “Bob”. It results in a file named foo.org.gpg.

The second line decrypts the foo.org.gpg file to foo2.org which will be identical to foo.gpg.

EPA - Emacs Interface to GPG

Emacs provides a better interface to GPG: Its EPA package enables me to encrypt/decrypt files in place. So I don’t have to keep jumping between the decrypted file (foo.org) and the encrypted file (foo.org.gpg) while working on it.

Below is the simple configuration that works well for me and its explanation.

 
(require 'epa-file)
(epa-file-enable)
(setq epa-file-encrypt-to "foo@bar.com")
(setq epg-pinentry-mode 'loopback)

epa-file-enable: is called to add hooks to find-file so that decrypting starts after opening a file in Emacs. It also ensures the encrypting starts when saving a GPG file I believe.

To stop this behaviour, call (epa-file-disbale) function.
epa-file-encrypt-to: to choose the default key for encryption.

This variable can be file specific, for example, to use the key belonging to foo2@bar.com key, drop the following in the file
```
;; -*- epa-file-encrypt-to: ("foo2@bar.com") -*-
```
epg-pinentry-mode: should be set to loopback so that GPG reads the password from Emacs’ minibuffer, otherwise, an external program (pinentry if installed) is used.

Org-Agenda and Dired

That’s more benefits Emacs offers in working with GPG files. Once I have the EPA configured, the org-agenda command works pretty well with encrypted files with no extra effort.

In the simplified example below, I have two GPG files as org-agenda-files. When the org-agenda is called, Emacs first try to decrypt the foo.org.gpg file. It requires me to type the password in a minibuffer.

The password will be cached by the GPG Agent and will be used to decrypt the bar.org.gpg assuming the same key is used for both files. So I only need to type the passphrase once.

 
(setq org-agenda-files '("foo.org.gpg" "bar.org.gpg"))
(org-agenda)

After that, org-agenda works as if these GPG files are normal unencrypted files; I can extract TODO lists, view the clock summary report, search text and check schedules/deadlines etc.

The dired provides functions to encrypt (shortcut “:e”) and decrypt (shortcut “:d”) multiple marked files in a dired buffer. Under the hood, they call the epa-encrypt-file and epa-decrypt-file functions.

Lisp to Close all GPG Files

It seems that once a buffer is decrypted upon opening or encrypted upon saving in Emacs, it stays as decrypted forever. So I need a utility function to close all the GPG buffers in Emacs to avoid leakage.

 
(defun yt/gpg--kill-gpg-buffers ()
  "It attempts to close all the file visiting buffers whose filename ends with .gpg.

It will ask for confirmation if the buffer is modified but unsaved."

  (kill-matching-buffers "\\.gpg$" nil t)
  )

Before I share my screens or start working in a coffee shop, I would call this function to ensure I close all buffers with sensitive data.

Footnotes

¹ stackexchange: symmetric vs asymmetric method

Jekyll in Emacs - Align URL with Headline

2023-12-19T00:00:00+00:00

Problem
Solution
Implementation

Problem

While I was working on improving the URL in my last post, I noticed the URLs are not readable, for example,

http://yitang.uk/2023/12/18/jekyll-in-emacs-update-blog-post-title-and-date/#org0238b9f

The URL links to the section called Code, so a much better URL should be

http://yitang.uk/2023/12/18/jekyll-in-emacs-update-blog-post-title-and-date/#Code

My notes show I have had this issue since 9 months ago. I made another attempt, but still could not find a solution!

Solution

I then switched to tidy up my Emacs configuration, and the variable org-html-prefer-user-labels caught my eye.

its documentation says

By default, Org generates its own internal ID values during HTML
export.

When non-nil use user-defined names and ID over internal ones.

So “#org0238b9f” is generated by org-mode. They are randomly generated; they change if I update the export file. It means every time I update a blog post, it breaks the URLs. This was a problem I wasn’t aware of.

Anyway, what’s important is that, in the end, it says

Independently of this variable, however, CUSTOM_ID are always
used as a reference.

That’s it, I just need to set CUSTOM_ID. That’s the solution to my problem. It is hidden in the documentation of some variables…

Implementation

So I need a function to loop through each node, and set the CUSTOM_ID property to its headline. The org-mode API provides three helpful functions for working with org files:

org-entry-get: to get a textual property of a node. the headline title is referenced as “ITEM”,
org-entry-put: to set a property of a node,
org-map-entries: to apply a function to each node.

I changed the final function a bit so it is used as an export hook (org-export-before-processing-functions) as an experiment. With this setup, it runs automatically whenever I export a blog post in org-mode to Markdown. Also, it works on the exported file so it leaves the original org file unchanged.

The code is listed below. It can also be found at my .emacs.d git repo which includes many other useful Emacs configurations for Jekyll.

 
 (defun yt/jekyll--create-or-update-custom_id-field ()
  "so that the CUSTOM_ID property is the same as the headline and 
the URL reflects the headline.

by default, the URL to a section will be a random number."
  (org-entry-put nil "CUSTOM_ID" (org-entry-get nil "ITEM"))
  )

(defun yt/jekyll--create-or-update-custom_id-field-buffer (backend)
  (when (eq backend 'jekyll-md)
    (org-map-entries 'yt/jekyll--create-or-update-custom_id-field)
    ))

(add-hook 'org-export-before-processing-functions 'yt/jekyll--create-or-update-custom_id-field-buffer)

Jekyll in Emacs - Update Blog Post Title and Date

2023-12-18T00:00:00+00:00

Emacs Lisp Time
Code

I’m the type of writer who writes first and comes up with the title later. The title in the end is usually rather different to what I started with. To change the title is straightforward - update the title and date fields in the front matter.

However, doing so leads to discrepancies between the title and date fields in front matter and the filename. In Jekyll, the filename consists of the original date and title when the post is first created.

This can be confusing sometimes in finding the file when I want to update a post. I have to rely on grep/ack to find the right files. A little bit of inefficiency is fine.

Recently, I realised that readers sometimes can be confused as well because the URL apparently also depends on the filename.

For example, I have my previous post in a file named 2022-12-08-trx-3970x.md. It indicates that I started writing it on 08 Dec with the initial title “trx 3970x”. A couple of days later on 13 Dec, I published the post with the title “How Much Does Threadripper 3970x Help in Training LightGBM Models?”.

The URL is however yitang.uk/2022/12/13/trx-3970x. It has the correct updated publish date, but the title is still the old one. This is just how Jekyll works.

Anyways, the correct URL should be

http://yitang.uk/2023/12/13/how-much-does-threadripper-3970x-help-in-training-lightgbm-models/

From that point, I decided to write a bit of Emacs Lisp code to help the readers.

Emacs Lisp Time

The core functionality is updating the filename and front matter to have the same publish date and title. It can breakdown into three parts:

when called, it promotes a new title. The publish date is fixed to whenever the function is called.
It renames the current blog post file with the new date and title. It also updates the title and date fields in the front matter accordingly.
It deletes the old file, closes the related buffer, and opens the new file so I can continue to work on it.

My Emacs Lisp coding skill is rusty but I managed to get it working in less than 2 hours. I won’t say it looks beautiful, but it does the job!

I spent a bit of time debugging, it turns out the (org-show-all) needs to be called first to flatten the org file, otherwise, editing with some parts of the content hidden can lead to unexpected results.

I always found working with the filename/directory in vanilla Emacs Lisp cumbersome, I wonder if is there any modern lisp library with a better API, something like Python’s pathlib module?

Code

Here are the main functions in case someone needs something similar. They are extracted from my Emacs configuration.

 
 (defun yt/jekyll-update-post-name ()
  "it update the post filename with a new title and today's date.

it also update the font matter."
  (interactive)
  (let* ((title (read-string "new title: "))
         (ext (file-name-extension (buffer-file-name)))  ;; as of now, the ext is always .org.

         ;; the new filename is in the format of {date}-{new-title}.org
         (filename (concat
                    (format-time-string "%Y-%m-%d-")
                    (file-name-with-extension (jekyll-make-slug title) ext)))

         ;; normalise the filename. 
         (filename (expand-file-name filename))

         ;; keep the current point which we will go back to after editing.
         (old-point (point))
         )
    (rename-file (buffer-file-name) filename) ;; update the filename
    (kill-buffer nil)  ;; kill the current buffer, i.e. the old file.
    (find-file filename)  ;; open the new file.
    (set-window-point (selected-window) old-point)  ;; set the cursor to where i was in the old file.

    ;; udpate title field. 
    ;; note jekyll-yaml-escape is called to ensure the title field is yaml friendly.
    (yt/jekyll-update-frontmatter--title (jekyll-yaml-escape title))    
    )

  )

(defun yt/jekyll-update-frontmatter--title (title)
  "Update the title field in the front matter.

title case is used. 
"
  (let* ((old-point (point)))

    ;; ensure expand all the code/headers/drawers before editing.
    (org-show-all)

    ;; go to the first occurence of 'title:'.
    (goto-char (point-min))
    (search-forward "title: ")

    ;; update the title field with the new title.
    (beginning-of-line)
    (kill-line)
    (insert (format "title: %s" title))

    ;; ensure the title is in title case
    (xah-title-case-region-or-line (+ (line-beginning-position) 7) (line-end-position))

    ;; save and reset cursor back to where it started.
    (save-buffer)    
    (goto-char old-point)
    ))

How Much Does Threadripper 3970x Help in Training LightGBM Models?

2023-12-13T00:00:00+00:00

Experiment Set up
i5-13600k - Efficient Cores Count
3970x - Disappointing Results
i5 vs. 3970x - Training in Parallel
CPU vs. GPU - Impressive Performance
Is the 3970x worth it?

Back in my Kaggle days, I always wondered how much my ranking could improve with a better computer. I finally pulled the triggers (twice) and got myself a 32-Cores Threadripper 3970x workstation.

Before I can tell if it helps my Kaggle competitions or not, I thought it would be interesting to quantify how much benefits I can get from upgrading the i5-13600k to 3970x in training LightGBM model.

The TLDR is:

The speedup is 3 times in training LightGBM using CPU.
To my surprise, it is 2 times faster using GTX 1080Ti GPU than i5-13600k.
There are no obvious gains from GTX 1080Ti to RTX 3080.

Experiment Set up

I use the data in the Optiver - Trading At The Close competition. There are about 500,000 rows and 100 features. I train a 3-fold (expanding window) LightGBM model. Repeating the same process with varying numbers of cores used in the process to get a performance graph like this:

Threadripper 3970x vs i5-13600k: Train LightGBM Models on CPU

i5-13600k - Efficient Cores Count

The i5-13600k has 6 performance cores and 8 efficient cores. In practice, I never use more than 6 cores in training ML models. My theory is mixing fast performance and slow efficient cores leads to a worse performance than using the performance cores alone. By specifying 6 cores, I assume the OS uses only performance cores.

The result shows that I was wrong - Using more than 6 cores can give considerable performance gain. It reduces the runtime by 10 minutes from 6 to 14 cores.

The only plausible explanation is that when training LightGBM with 6 cores, it is already mixed with efficient cores. Therefore I see an increases in performance while adding more cores.

Regardless I will start to use 12 cores in practise.

3970x - Disappointing Results

I know the performance gain will not scale linearly with the number of cores but I wasn’t expecting that adding more cores can slow down the model training.

The graph shows the 3970x achieves its best performance at using 12 cores. After that, adding more cores increases the runtime.

This type of behaviour is usually observed in simple tasks where the overhead of coordinating between cores outweighs the benefits of extra cores bring in.

But training thousands of decision trees with half a million data points is definitive not in this simple task category. So I don’t understand why this is happening.

i5 vs. 3970x - Training in Parallel

For 6 cores, it took i5 51 minutes and 3970x 42 minutes, which is about 1.2 speedup which is not bad. The same speed boost is also observed at using 10 and 12 cores.

I found this consistent speedup confusing because there’s a mix of performance and efficient cores in i5, so in theory every performance core I add in 3970x should increase the performance marginal when compared to i5.

In general, because of the poor scalability with respect to the number of cores, the best performance is achieved when training the model with a small number of cores and running multiple training in parallel. This is the trick I use to get the extra performance boost for CPU-bound tasks.

Here’s the setup for each computer:

i5-13600: use 6 cores to train each model, and train 2 models in parallel. They are 2 cores left for OS background activities.
3970x: also use 6 cores to train each model, but train 5 models in parallel! It also leaves 2 cores for OS background activities.

After a little bit of maths, it takes 14 hours for 3970x to train 100 models, and 42.8 hours for i5, so the speedup is 3 times. This is just based on my theory. It would be good to actually run the experiment and see the actual numbers.

Table 1: Training 100 models in parallel setting.
CPU	Runtime of 1 model (S)	No. models in Parallel	No. Batches	Total Runtime (H)
13600k	3083	2	50	42.8
3970x	2523	5	20	14.0

So the most benefit I can get from 3970x is in running multiple experiments in parallel!

CPU vs. GPU - Impressive Performance

I have a GTX 1080Ti in my i5 PC for running deep learning models and CUDA code. I never use it for LightGBM because the GPU implementation was slower than the CPU in 2019 when I tried it.

In summer Guolin Ke, the author LightGBM, promised a significant improvement in GPU performance when he was looking for volunteers to work on improving LightGBM’s GPU algorithm.

Since I have the experiments set up already, it took me little time to repeat the same experiments using the GPU trainer. All I did was adding device_type=’gpu’ in the configuration files.

Table 2: Runtime of training a single mdoel
# CPU Cores	i5-13600k	tr-3970x	GTX 1080ti	RTX 3080
6	3083	2523	1435	1256
10	2695	1940	1269	1147

The result shocks me: I can get 2 times speedup just by switching from i5 to 1080Ti with one additional line in the config and it outperforms the 3970x in training single model setting by a big margin!

Is the 3970x worth it?

I found myself asking this question after seeing the results. In the context of this experiment, no, it makes no sense to spend £2,000 to get 3 times speedup when I can simply switch to 1080Ti to get 2 times speed up with no costs.

However, the reason I go for the Threadripper and the TRX40 platform is the 128 PCIe 4.0 lanes. The workstation is capable of running 4 GPUs at the same time at full capability while as i5 can only run 1 GPU.

If I had 4 GTX 3080 installed, it would finish training 100 models in just under 8 hours! That’s 5.25 speedup to i5 and 1.75 speedup to 3970x in parallel setting.

This calculation is not for just entertainment. It turns out that utilising multiple GPU to train gradient boost tree can be a really big thing!

I just found another reason to buy more GPUs! :)

State of This Blog

2023-03-28T00:00:00+01:00

Technical Debt
Revisit the Tech Stack
Long Ride with Jekyll

This static blog is built using Jekyll in 2014. It survived after 7 years which is a success when it comes to personal blogging. Part of the reason is having a good blogging workflow: write posts in Org Mode, export to HTML with a front matter, build the site using Jekyll, send the folder to an Amazon S3 bucket, and that’s it. All done in Emacs of course.

Technical Debt

I added a few things to the workflow to enhance the reading experience including code highlights, centred images with caption, table of content etc. There are more features I want to add but at the same time, I want to be able to just write.

With that mindset, whenever there are issues, I apply quick fixes without a deep understanding of the actual causes. It seems efficient until recently some fixes become counter-productive.

I started seeing underscore (_) is exported as \_ and <p> tag appears in code snippets. It all sounds like quick fix, but I just couldn’t get it correct after few hours. For the last few posts, I had to manually fix them for each of the read-edit-export-fix iteration.

Revisit the Tech Stack

I have an ambitious goal for this blog. So it is time to go sweep the carpet. I studied the technologies used for this blog, Jekyll, AWS and Org Mode exporting. It was a good chance to practise Org-roam for taking atomic notes. The time is well spent as I learnt a lot.

I was impressed I got the whole thing up and running 7 years ago. I don’t think I have the willpower to do it now.

Still, there are a lot of things that I do not have a good understand, e.g. the Liquid templates, HTML and CSS tags etc. The syntax just puts me off.

Long Ride with Jekyll

I prefer a simple format like Org Mode or Markdown and don’t have to deal with HTML/CSS at all. There are a couple of occasions when I cannot resist the temptation to look for an alternative to Jekyll. There’s no luck in the search. It seems HTML is the only way because it is native to the web.

So the plan is to stick with Jekyll for at least a few years. In the next couple of weeks, I’d try to fix all the issues, after that, gradually add more features to enhance the writing and reading experience.

I hope people who also uses the similar tech stack (Org-mode, Emacs, Jekyll, AWS) can benefit my work.

Setup Emacs Servers in MacOS

2023-02-09T00:00:00+00:00

Emacs Server Configuration
Launch Emacs GUI in Terminal
Launch Emacs GUI in Spotlight

I switched to MacOS last year for editing home gym videos. I was and am still amazed by how fast the M1 chip is for exporting 4K videos. The MacOS also enriched the Emacs experience which makes it deserve another blog post.

So I have been slowly adapting my Emacs configuration and workflow to MacOS. One of the changes is the Emacs server.

The goal is to have fully loaded Emacs instances running all the time so I can use them at any time and anywhere, in Terminal or Spotlight. They are initiated upon login. In cases of Emacs crashes (it is rare but more often than I like) or I have to stop them because I messed up the configuration, they restart automatically.

Emacs Server Configuration

I have this setup in Linux using systemd, as in my previous blog post.

In MacOS, the launchctl is the service manager. It provides a user interface to list, start and stop services.

To build an Emacs server, create a plist file in ~/Library/LaunchAgents folder. In my case, I named it emacs_work.plist.

# cat ~/library/LaunchAgents/emacs_work.plist
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>emacs_work</string>
    <key>ProgramArguments</key>
    <array>
      <string>/opt/homebrew/opt/emacs-plus@31/bin/emacs</string>
      <string>--fg-daemon=work</string>
      <string>--init-directory=~/.config/emacs/emacs.d_v31</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>    
    <key>StandardOutPath</key>
    <string>/tmp/emacs_work.stdout.log</string>
    <key>StandardErrorPath</key>
    <string>/tmp/emacs_work.stderr.log</string>
  </dict>
</plist>

It is an extension of Emacs Plus’ plist file. I made a few changes for running two Emacs servers: one for work (data sciences, research) and one for personal usage (GTD, books). Taking the “work” server as an example, the important attributes of the plist configuration file are:

Line 5: The unique service name to launchctl
Line 8: The full path to the Emacs program. In my case, it is /opt/homebrew/opt/emacs-plus@31/bin/emacs
Line 9: The “–fg-daemon” option set the Emacs server name to “work”. Later I can connect to this server by specifying “-s=work” option to emacsclient
Line 13: The KeepAlive is set to true so it keeps trying to restart the server in case of failures
Line 16 and 18: The location of standard output and error files. They are used to debug. Occasionally I have to check those files to see why Emacs servers stopped working, usually because of me introducing bugs in my .emacs.d.

With the updated plist files in place, I start the Emacs servers with

 
launchctl load -w ~/Library/LaunchAgents/emacs_work.plist
launchctl load -w ~/Library/LaunchAgents/emacs_org.plist

The launchctl list | grep -i emacs is a handy snippet that lists the status of the services whose name includes “emacs”. The output I have right now is

PID	Exit Code	Server ID
1757	0	emacs_org
56696	0	emacs_work

It shows both Emacs servers are running fine with exit code 0.

Launch Emacs GUI in Terminal

I can now open a Emacs GUI and connect it to the “work” Emacs server by running emacsclient -c -s work &. The -c option

Launch Emacs GUI in Spotlight

In MacOS, I found it is natural to open applications using Spotlight, for example, type ⌘ + space to invoke Spotlight, put “work” in the search bar, it narrows the search down to “emacs_work” application, and hit return to finalise the search. It achieves the same thing as the command above but can be used anywhere.

I uploaded a demo video on YouTube to show it in action. You might want to watch it at 0.5x speed because I typed so fast…

To implement this shortcut, open “Automator” application, start a new “Application”, select “Run Shell Script”, and paste the following bash code

 
/opt/homebrew/opt/emacs-plus@31/bin/emacsclient \
    --no-wait \
    --quiet \
    --suppress-output \
    --create-frame -s work \
    "$@"

and save it as emacsclient_work in the ~/Application folder.

Essentially, the bash script above is wrapped up as a MacOS application, named emacsclient_work and the Spotlight searches the application folder by default.

Speed Up Sparse Boolean Data

2023-01-05T00:00:00+00:00

Imaging On-the-fly
Scipy Sparse Matrix
Numpy Bites
Problems of Having Millions of Files

I’m working on replicating the (Re-)Imag(in)ing Price Trends paper - the idea is to train a Convolutional Neutral Network (CNN) "trader" to predict the stocks' return. What makes this paper interesting is the model uses images of the pricing data, not in the traditional time-series format. It takes financial charts like the one below and tries to mimic the traders' behaviours to buy and sell stocks to optimise future returns.

Alphabet 5-days Bar Chart Shows OHLC Price and Volume Data

I like this idea. So it becomes my final assignment for Deep Learning Systems: Algorithm and Implementations course.

Imaging On-the-fly

To train the model, the price and volume data are transformed into black-white images which is just a 2D matrix with 0s and 1s. For just around 100 stocks' pricing history, there are around 1.2 million images in total.

I used the on-the-fly imaging process during training: in each batch, it loads pricing data for a given stock, sample one day in the history, slice a chunk of pricing data, and then convert it to an image. It takes about 0.2 milliseconds (ms) to do all that, so in total it takes 4 minutes to loop through all the 1.2 million images.

%%timeit 
df = MarketData(DATA_DIR)['GOOGL']
imager = ImagingOHLCV(img_resolution, price_prop=price_prop)
img = imager(df.tail(5))

1.92 ms ± 26.9 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

To train 10 epochs, that's 40 minutes in loading data. To train one epoch on the full dataset with 5,000 stocks, that's 200 minutes in loading data alone!

PyToch utilises multiple processing in loading the data using CPU while training using GPU. So the problem is less severe, but I'm using the needle, the deep learning framework we developed during the course, it does have this functionality yet.

During training using needle, the GPU utilisation is only around 50%. After all the components in the end-to-end are almost completed, it is time to train with more data, go deeper (larger/more complicated morel), try hyper-parameters tuning etc.

But before moving to the next stage, I need to improve the IO.

Scipy Sparse Matrix

In the image above, there are a lot of black pixels or zeros in the data matrix. In general only 5%-10% of pixels are white in this dataset.

So my first attempt was to use scipy's spare matrix instead of numpy's dense matrix: I save the sparse matrix, loaded it, and then convert it back to a dense matrix for training CNN model.

%%timeit
img_sparse = sparse.csr_matrix(img)
sparse.save_npz('/tmp/sparse_matrix.npz', img_sparse)
img_sparse_2 = sparse.load_npz('/tmp/sparse_matrix.npz')
assert np.all(img_sparse_2 == img)

967 µs ± 4.99 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

It reduces the IO time to 1ms, so about half of the time, not bad, but I was expecting a lot more as the sparseness is high.

Numpy Bites

Then I realised the data behind images is just 0 and 1, in fact, a lot of zeros, and only some are 1. So I can ignore the 0s and only need to save those 1s, then reconstruct the images using those 1.

It is so simple that numpy has functions for this type of data processing already. The numpy.packbites function converts the image matrix of 0 and 1 into a 1D array whose values indicate where the 1s are. Then the numpy.unpackbits does the inverse: it reconstructs the image matrix by using the 1D location array.

This process reduces the time of loading one image to 0.2 milliseconds, that's 10 times faster than the on-the-fly method with only a few lines of code.

%%timeit 
temp_file = "/tmp/img_np_bites.npy"
img_np_bites = np.packbits(img.astype(np.uint8))
np.save(temp_file, img_np_bites)
img_np_bites = np.load(temp_file)
img_np_bites = np.unpackbits(img_np_bites).reshape(img.shape)
assert np.all(img_np_bites == img)

194 µs ± 3.95 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

Another benefit is the file size is much smaller: it is 188 bytes compared to 1104 bytes using sparse matrix. So it takes only 226MB of disk space to save 1.2 million images!

Path('/tmp/img_np_bites.npy').stat().st_size, Path('/tmp/sparse_matrix.npz').stat().st_size

188, 1104

Problems of Having Millions of Files

It takes a couple of minutes to generate 1.2 million files on my Debian machine. It is so quick! But then I release this approach is not scalable without modification because there's a limited number of files the OS can accommodate. The technical term is Inode. According to this StackExchange question, once the filesystem is created, one cannot increase the limit (Yes, I was there).

Without going down to the database route, one quick workaround is to bundle the images together, for example, 256 images in one file. So later in training, load 256 images in one go, then split them into chunks. Just ensure the number of images is a multiple of the batch size used in training so I don't have to deal with unequal batch sizes. Since those bundled images are trained together, it reduces the randomness of SGD, so I won't bundle too many images together, 256 sounds about right.

The LSP and other tools can cause problems when they are monitoring folders with a large number of files. Moving them out of the project folder is the way to go so Emacs won't complain or freeze.

PoorMan's CI in Emacs

2022-12-16T00:00:00+00:00

I have been working on the Deep Learning System course. It is the hardest course I ever studied after university. I would never thought that I need CI for a personal study project. It just shows how complex this course is.

Here is the setup: the goal is to develop a pytorch-like DL library that supports ndarray ops, autograd, and to implement DL models, LSTM for example, from scratch. That's the exciting math part. The tricky part is it supports both CPU devices with C++11 and GPU devices with Cuda. On the user front, the interface is written in Python. I worked on my M1 laptop most of the time, and switch to my Debian desktop for Cuda implementation.

It was a fine Saturday afternoon, I made a breakthrough in implementing the gradient of Convolution Ops in Python after couple of hours of tinkering in a local coffee shop. I rushed home, boosted up Debian to test the Cuda backend, only to find "illegible memory access" error!

It took me a few cycles of rolling back to the previous change in git to find where the problems are. It made me think about the needs of CI. In the ideal scenario, I would have a CI that automatically runs the tests on the CPU and Cuda devices to ensure one bug-fix on CPU side doesn't introduce new bugs on the Cuda, and vice versa. But I don't have this setup at home.

Two Components of PoorMan CI

So I implemented what I call PoorMan CI. It is a semi-automated process that gives me some benefits of the full CI. I tried hard to refrain from doing anything fancy because I don't have time. The final homework is due in a few days. The outcome is simple yet powerful.

The PoorMan CI consists of two parts:

a bunch of bash functions that I can call to run the tests, capture the outputs, save them in a file, and version control it

For example, wrap the below snippet in a single function

pytest -l -v -k "not training and cuda" \
       > test_results/2022_12_11_12_48_44__fce5edb__fast_and_cuda.log
git add test_results/2022_12_11_12_48_44__fce5edb__fast_and_cuda.log

a log file where I keep track of the code changes, and if the new change fixes anything, or breaks anything.

In the example below, I have a bullet point for each change committed to git with a short summary, and a link to the test results. The fce5edb and f43d7ab are the git commit hash values.
```
- fix grid setup, from (M, N) to (P, M)!
[[file:test_results/2022_12_11_12_48_44__fce5edb__fast_and_cuda.log]]

- ensure all data/parameters are in the right device. cpu and cuda, all pass! milestone.
[[file:test_results/2022_12_11_13_51_22__f43d7ab__fast_and_cuda.log]]
```

As you can see, it is very simple!

Benefits

It changed my development cycle a bit: each time before I can claim something is done or fixed, I run this process which takes about 2 mins for two fast runs. I would use this time to reflect on what I've done so far, write down a short summary about what's got fixed and what's broken, check in the test results to git, update the test log file etc.

It sounds tedious, but I found myself enjoying doing it, it gives me confidence and reassurance about the progress I'm making. The time in reflecting also gives my brain a break and provides clarity on where to go next.

During my few hours of using it, it amazes me how easy it is to introduce new issues while fixing existing ones.

Implement in Org-mode

I don't have to use Org-mode for this, but I don't want to leave Emacs :) Plus, Org-mode shines in literate programming where code and documentation are put together.

This is actually how I implemented it in the first place. This section is dedicated to showing how to do it in Org-mode. I'm sure I will come back to this shortly, so it serves as documentation for myself.

Here is what I did: I have a file called poorman_ci.org, a full example can be found at this gist. An extract is demonstrated below.

I group all the tests logistically together into "fast and cpu", "fast and cuda", "slow and cuda", "slow and cuda". I have a top level header named group tests, Each group has their 2nd-level header.

The top header has a property drawer where I specify the shell session within which the tests are run so that

* grouped tests
:PROPERTIES:
:CREATED:  [2022-12-10 Sat 11:32]
:header-args:sh:    :session *hw4_test_runner* :async :results output :eval no
:END:

it is persistent. I can switch to the shell buffer named hw4_test_runner and do something if needed
it runs asynchronically on the background

All the shell code block under the grouped tests inherits those attributes.

The first code block defines variables that used to create a run id. It uses the timestamp and the git commit hash value. The run id is used for all the code blocks.

#+begin_src sh :eval no
wd="./test_results/"
ts=$(date +"%Y_%m_%d_%H_%M_%S")
git_hash=$(git rev-parse --verify --short HEAD)
echo "run id: " ${ts}__${git_hash}$
#+end_src

To run the code block, move the cursor inside the code block, and hit C-c C-c (control c control c).

Then I define the first code block to run all the tests on CPU except language model training. I name this batch of tests "fast and cpu".

#+begin_src sh :var fname="fast_and_cpu.log"
fname_full=${wd}/${ts}__${git_hash}__${fname}
pytest -l -v -k "not language_training and cpu" \
     2>&1 | tee ${fname_full}
#+end_src

It creates the full path of the test results. The fname variable is set at the code clock header, this is a nice feature of Org-mode.
pytest provides an intuitive interface for filtering tests, here I use "not language_training and cpu".
The tee program is used to show the outputs and errors and at the same time save them to a file.

Similarly, I define code blocks for "fast and cuda", "slow and cpu", "slow and cuda".

So at the end of the development cycle, I open the poorman_ci.org file, run the code blocks sequentially, and manually update the change log. That's all.

Machine Learning in Emacs - Copy Files from Remote Server to Local Machine

2022-07-31T00:00:00+01:00

dired-rsync is a great additional to my Machine Learning workflow in Emacs

File Manager GUI
Rsync Tool in CLI
Emacs’ Way: dired-rsync
Setup for dired=rsync
Enhance dired-rsync with compilation mode

For machine learning projects, I tweaked my workflow so the interaction with remote server is kept as less as possible. I prefer to do everything locally on my laptop (M1 Pro) where I have all the tools for the job to do data analysis, visualisation, debugging etc and I can do all those without lagging or WI-FI.

The only usage of servers is running computation extensive tasks like recursive feature selection, hyperparameter tuning etc. For that I ssh to the server, start tmux, git pull to update the codebase, run a bash script that I prepared locally to fire hundreds of experiments. All done in Emacs of course thanks to Lukas Fürmetz’s vterm.

The only thing left is getting the experiment results back to my laptop. I used two approaches for copying the data to local: file manager GUI and rsync tool in CLI.

Recently I discovered dired-rsync that works like a charm - it combines the two approaches above, providing a interactive way of running rsync tool in Emacs. What’s more, it is integrated seamlessly into my current workflow.

They all have their own use case. In this post, I brief describe those three approaches for coping files with a focus on dired-rsync in terms of how to use it, how to setup, and my thoughts on how to enhance it.

Note the RL stands for remote location, i.e. a folder a in remote server, and LL stands for local location, the RL’s counterpart. The action in discussion is how to efficiently copying files from RL to LL.

File Manager GUI

This is the simplest approach requires little technical skills. The RL is mounted in the file manager which acts as an access point so it can be used just like a local folder.

I usually have two tabs open side by side, one for RL, and one for LL, compare the differences, and then copy what are useful and exists in RL but not in LL.

I used this approach on my Windows work laptop where rsync is not available so I have to copy files manually.

Rsync Tool in CLI

The rsync tool is similar to cp and scp but it is much more power:

It copies files incrementally so it can stop at anytime without losing progress
The output shows what files are copied, what are remaining, copying speed, overall progress etc
Files and folders can be included/excluded by specifying patterns

I have a bash function in the project’s script folder as a shorthand like this

copy_from_debian_to_laptop () {
    # first argument to this function
    folder_to_sync=$1
    # define where the RL is 
    remote_project_dir=debian:~/Projects/2022-May
    # define where the LL is 
    local_project_dir=~/Projects/2022-May          
    rsync -avh --progress \
	  ${remote_project_dir}/${folder_to_sync}/ \
	  ${local_project_dir}/${folder_to_sync}
}

To use it, I firstly cd (change directory) to the project directory in terminal, call copy_from_debian_to_laptop function, and use the TAB completion to quickly get the directory I want to copy, for example

copy_from_debian_to_laptop experiment/2022-07-17-FE

This function is called more often from a org-mode file where I kept track of all the experiments.

Emacs’ Way: `dired-rsync`

This approach is a blend of the previous two, enable user to enjoy the benefits of GUI for exploring and the power of rsync.

What’s more, it integrates so well into the current workflow by simply switching from calling dired-copy to calling dired-rsync, or pressing r key instead of C key by using the configuration in this post.

To those who are not familiar with copying files using dired in Emacs, here is the step by step process:

Open two dired buffer, one at RL and one at LL, either manually or using bookmarks
Mark the files/folders to copy in the RL dired buffer
Press r key to invoke dired-rsync
It asks for what to copy to. The default destination is LL so press Enter to confirm.

After that, a unique process buffer, named *rsync with a timestamp suffix, is created to show the rsync output. I can stop the copying by killing the process buffer.

Setup for `dired=rsync`

The dired-rsync-options control the output shown in the process buffer. It defaults to “-az –info=progress2”. It shows the overall progress in one-line, clean and neat (not in MacOS though, see Issue 36). Sometimes I prefer “-azh –progress” so I can see exactly which files are copied.

There are other options for showing progress in modeline (dired-rsync-modeline-status), hooks for sending notifications on failure/success (dired-rsync-failed-hook and dired-rsync-success-hook).

Overall the library is well designed, and the default options work for me, so I can have a bare-minimal configuration as below (borrowed from ispinfx):

(use-package dired-rsync
  :demand t
  :after dired
  :bind (:map dired-mode-map ("r" . dired-rsync))
  :config (add-to-list 'mode-line-misc-info '(:eval dired-rsync-modeline-status 'append))
  )

There are two more things to do on the system side:

In macOS, the default rsync is a 2010 version. It does not work with the latest rsync I have on Debian server so I upgrade it using brew install rsync.
There no way of typing password as a limitation of using process buffer so I have to ensure I can rsync without remote server asking for password. It sounds complicated but fortunately it takes few steps to do as in Setup Rsync Between Two Servers Without Password.

Enhance `dired-rsync` with compilation mode

It’s such a great library that makes my life much easier. It can be improved further to provide greater user experience, for example, keep the process buffer alive as a log after the coping finished because the user might want to have a look later.

At the moment, there’s no easy way of changing the arguments send to rsync. I might want to test a dry-run (adding -n argument) so I can see exactly what files are going to be copied before running, or I need to exclude certain files/folders, or rerun the coping if there’s new files generated on RL.

If you used compilation buffer before, you know where I am going. That’s right, I am thinking of turning the rsync process buffer into compilation mode, then it would inherit these two features:

Press g to rerun the rsync command when I know there are new files generated on the RL
Press C-u g (g with prefix) to change the rsync arguments before running it for dry-run, inclusion or exclusion

I don’t have much experience in elisp but I had a quick look at source code, it seems there’s no easy of implementing this idea so something to add to my ever-growing Emacs wish-list.

In fact, the limitation comes from using lower level elisp functions. The Emacs Lisp manual on Process Buffers states that

Many applications of processes also use the buffer for editing input to be sent to the process, but this is not built into Emacs Lisp.

What a pity. For now I enjoy using it and look for opportunities to use it.

Move Between Windows in Emacs using windmoveMD

2022-07-05T00:00:00+01:00

Started Seeing
Ace-Window
Windmove
Back to where it started

Started Seeing

The good thing about Emacs is that you can always tweak it to suit your needs. For years I’ve been doing it for productivity reasons. Now for the first time, I’m doing it for health reasons.

Life can be sht sometimes, when I was in my mid 20s, I was reshaping every aspects of my life for good. But optician told me my vision can only get worse. I wasn’t paying much attention, busy with my first job and learning.

Last month, I was told my right eye’s vision got whole point worse, whatever that means. Now I’m wearing a new pair of glasses, seeing the world in 4K using both eyes, noticing so much details. It makes the world so vibrate and exciting. It comes with a price though, my eyes get tired quickly, and it become so easy to get annoyed by little things.

One of them is switching windows in Emacs. Even though I am in the period of calibrating to the new glasses, I decided to take some actions.

Ace-Window

Depends on the complexity of the tasks, I usually have about 4-8 windows laid on my 32 inch monitor. If that’s not enough, I would have an additional frame of similar windows layout, doubling the number of windows to 8-16.

So I found myself switching between windows all the time. The action itself is straightforward with ace-window.

The process can be breakdwon into five steps:

Invoke ace-window command by pressing F2 key,
The Emacs buffers fade-in,
A red number pops-up at the top left corner of each window,
I press the number key to switch the window it associates with,
After that, the content in each Emacs buffer are brought back.

This gif from ace-window git repo demonstrates the process.

This approach depends on visual feedback - I have to look at the corner of the window to see the number. Also, the screen flashes twice during the process.

I tried removing the background dimming, increase the font size of the number to make it easier to see, and bunch of other tweaks.

In the end, my eyes were not satisfied.

Windmove

So I started looking for alternative approaches and found windmove which is built-in.

The idea is simple - keep move to the adjacent window by move left, right, up, or down until it arrives at the window I want.

So it uses the relative location between windows instead of assigning each window a unique number and then using the number for switching.

Is it really better? Well with this approach, I use my eyes a lot less as I do not have to look for the number. Plus, I feel this is more nature as I do not need to work out the directions, somehow I just know I need to move right twice or whatever to get to the destination.

The only issue I had so far is the conflicts with org-mode’s calendar. I like the keybinding in org-mode, so I disabled windmove in org-mode’s calendar with the help from this stackoverflow question.

The following five lines of code is all I need to use windmove.

(windmove-default-keybindings)
(define-key org-read-date-minibuffer-local-map (kbd "<left>") (lambda () (interactive) (org-eval-in-calendar '(calendar-backward-day 1))))
(define-key org-read-date-minibuffer-local-map (kbd "<right>") (lambda () (interactive) (org-eval-in-calendar '(calendar-forward-day 1))))
(define-key org-read-date-minibuffer-local-map (kbd "<up>") (lambda () (interactive) (org-eval-in-calendar '(calendar-backward-week 1))))
(define-key org-read-date-minibuffer-local-map (kbd "<down>") (lambda () (interactive) (org-eval-in-calendar '(calendar-forward-week 1))))

I created an git branch for switching from ace-window to windmove. I would try it for a month before merge it into master branch.

Back to where it started

After using it for few days, I realised this is the very package I used for switch windows back in 2014 when I started learning Emacs. I later then switched to ace-window because it looks pretty cool.

Life is changing, my perspectives are changing, so is my Emacs configuration. This time, it goes back to where I started 8 years ago.

Wireless Backup Solution Using Raspberry Pi for MacOS

2022-03-11T00:00:00+00:00

If you need automated backups for Time Machine and have a Raspberry Pi, You will find this post useful.

Motivation
Wireless Backup Solution
Set Up Raspberry Pi
Time Machine Backup frequency
Backup for Backups

Motivation

After 3 months using my brand new MacBook Pro 14 M1 Pro, one of the USB-C port stopped working. I will have to send it back, not sure what Apple will do with it but I can't bear the risk of losing data. So I need a backup.

In fact, I need to backup regularly for situation like this so that's why I worked on it.

Wireless Backup Solution

The easiest solution is to get a USB-C portable SSD, plug it into my laptop and open Time Machine to start back up, do it once a week and call it a day.

But I'm reluctant to add more devices to my already cluttered home lab. There are a few hard drives in the drawers, it would be good to utilise them.

So I decided to set up a Time Machine backup solution using on my Raspberry Pi 4. The benefits are

no additional costs, save me about £50-£100
no need to buy new stuff, so fewer things to care of
wireless backup to keep my desk clean

Later I realised the benefits of having a wireless backup is overlooked. It can backup anytime and anywhere in my house. Also, because of convenience, I can have more granular backups - instead of weekly backup, I have hourly backup without getting the cables and hard drives. I do less but get more value out of it.

The only concern I had was the speed. It turns out with SAMBA 3 protocol, I can get 55 MB/s write speed and 40 MB/s read speed from laptop to Raspberry Pi. So in theory, it would take around 2.5 hours to backup my 500 GB laptop. It might be a lot but only for the first backup, the subsequent incremental backup would be much simpler and faster, for example, as of now, the Time Machine completed a new backup within 3 minutes in the background without my notice.

A portal USB-C SSD can finish the backup within minutes but it's an overkill for an ordinary user like me and it's inconvenient.

So I'm satisfied with the current solution.

Set Up Raspberry Pi

I read a few guides on setting up Raspberry Pi for Time Machine, and I found this guide most accurate and useful.

One thing I noticed is the AFP (Apple File Protocol) is deprecated, so make sure you use SAMBA as the protocol.

Additionally, I followed this stack overflow answer to auto-mount the SAMBA server so that every time I reboot my laptop, the Time Machine will be ready to back up.

Time Machine Backup frequency

By default, Time Machine does hourly backup.

If you feel hourly backup is not necessary, you can change it by updating this file

/System/Library/LaunchDaemons/com.apple.backupd-helper.plist

for example, to change the frequency from hourly to daily backup, change the interval value from 3600 to 43200.

In the end, I left it with the default hourly backup so it does many small backup hourly instead of one big backup daily.

Backup for Backups

After couple of hours of work, I managed to get a wireless backup solution for my laptop so I won't have to worry about data loss. Plus I can time-travel files at hourly intervals.

One concern that occurred to me was the backup sits on my local hard drive. If the hard drive died, I would lose all my backups.

To solve that problem, I will have to go through the rabbit hole of doing backup for backups, or backup to a remote location or cloud, or setup a Raspberry Pi RAID.

At the moment, I'm not very concerned - I have Apple iCloud to back up my photos, videos, notes etc and I use GitHub to host my org-files and code. So having a backup for backups is not necessary for me for now.

Managing Emacs Server as Systemd Service

2021-06-18T00:00:00+01:00

Using Emacs Server Without Systemd
How to Implement As Systemd Service
Enhance User Experience
- sudo Privilege
- Environment Variables
Start Emacs Server Before Login?

Using Emacs Server Without Systemd

I live in Emacs entirely apart from using browser for googling. Having an Emacs server running on the background makes Emacs available all the time. So I won't worry about closing it accidental.

It is not hard to do that, just run

emacs --daemon

in command line to start the Emacs server. It will load user configuration file as usual. Then run

emacsclient -c &

to open an Emacs GUI instance that uses the Emacs server. That's how I have been doing for a while.

An better approach is using systemd. It is the services manager of Linux. Whenever my Debian 11 laptop boot up, systemd would start a bunch of services in parallel, for example, Networking manager connects WIFI, Bluetooth connects wireless keyboard so everything would be ready after I login. And I want Emacs to be ready as well.

I can achieve that by simply having an shell script automatically running after login. But there are benefits of using systemd. It has bunch of sub-commands for managing services, for example, checking logs, status etc.

It's a nice tool to have, I can use it for example Jupyter Notebook server.

That's why I pulled the trigger and spent 2 hours in implementing and testing it. Here's the technical bit.

How to Implement As Systemd Service

In order to use systemd to manage Emacs server, I firstly need a configuration file (which is called unit file). Debian Wiki provides a short description of the syntax and parameter of unit file.

I found an simple one in Emacs Wiki. It looks like this

[Unit]
Description=Emacs text editor
Documentation=info:emacs man:emacs(1) https://gnu.org/software/emacs/

[Service]
Type=forking
ExecStart=/usr/bin/emacs --daemon
ExecStop=/usr/bin/emacsclient --eval "(kill-emacs)"
Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket
Restart=on-failure

[Install]
WantedBy=default.target

The important parameters are

ExecStart: It tells systemd what to do when starting Emacs

service, in this case it runs /usr/bin/emacs --daemon command.

ExecStop: it tells systemd what to do when shutting down Emacs

service, in this case it runs /usr/bin/emacsclient --eval "(kill-emacs)" command.

If you are using an Emacs built in a difference directory, you have to change /usr/bin/emacs to wherever your Emacs is located.

Then save the configuration file as ~.config/systemd/user/emacs.service/.

After that run

systemctl enable --user emacs

so systemd would copy the configuration file into central places and it would start Emacs service at boot time.

To run Emacs service right now, use

systemctl start --user emacs

This is what I see in my console

emacs.service - Emacs text editor
Loaded: loaded (/home/yitang/.config/systemd/user/emacs.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-06-14 09:12:26 BST; 24h ago
Docs: info:emacs
man:emacs(1)
https://gnu.org/software/emacs/
Main PID: 5222 (emacs)
Tasks: 5 (limit: 19027)
Memory: 154.7M
CPU: 3min 25.049s
CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/emacs.service
├─ 5222 /usr/bin/emacs --daemon
└─16086 /usr/bin/aspell -a -m -d en_GB -p /home/yitang/git/.emacs.d/local/ispell-dict --encoding=utf-8

Jun 14 09:11:57 7270 emacs[5222]: No event to add
Jun 14 09:11:57 7270 emacs[5222]: Package dash-functional is obsolete; use dash 2.18.0 instead
Jun 14 09:12:01 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/config/org-mode.el (source)...done
Jun 14 09:12:01 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/config/refile.el (source)...
Jun 14 09:12:01 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/config/refile.el (source)...done
Jun 14 09:12:01 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/config/scripting.el (source)...
Jun 14 09:12:26 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/config/scripting.el (source)...done
Jun 14 09:12:26 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/load_config.el (source)...done
Jun 14 09:12:26 7270 emacs[5222]: Starting Emacs daemon.
Jun 14 09:12:26 7270 systemd[4589]: Started Emacs text editor.

Enhance User Experience

For far, I have the following two tweaks to make the usage of systemd more pleasant.

sudo Privilege

The Emacs server is started using my own account, so it doesn't have the sudo privilege. In order to edit files that requires sudo permission, simple open the file in Emacs, or in command line with

emascclient -c FILENAME

then type M-x sudo inside Emacs, type the sudo password. If the password is correct, I can edit and save the file as sudo user.

Environment Variables

The customised shell configuration in .bashrc are loaded when opening an interactive shell session. So the Emacs server managed by systemd would not have the environment variables, alias, functions or whatever defined in .bashrc.

This stackoverflow post provides the rationale and how to tweak the unit file so systemd would load .bashrc.

This problem can solved a lot easier on the Emacs side, by using exec-path-from-shell package. It will ensure the environment variables inside Emacs are the same as in the user's interactive shell.

Simply put the following in your .emacs would do the trick.

(exec-path-from-shell-initialize)

Start Emacs Server Before Login?

The systemd services under my account would only start after I login. Because I have tons of Emacs configuration, I still have to wait few seconds before Emacs server is ready. So it would be awesome to have the Emacs server starting to load before I login.

This doesn't seems to be simple to implement, because technically, it would require the Emacs server to be defined on system level, but it will load files in my personal home drive without me being logged in. It might be still okay since I'm the sole user of my laptop, but I have to tweak the permissions and would probably end up with non-secure permission setting.

So I leave this idea here.

Kaggle Avito Demand Prediction Challenge - 22th Solution

2018-07-01T00:00:00+01:00

Final Ensemble Models
Collaboration
Some Kagglers to Avoid

Avito Demand Prediction Challenge asks Kagglers to predict the "demand" likelihood of an advertisement. If an listed 2nd-hand Iphone 6 is selling for £20,000, then the "demand" is likely to be very low. This is the my first competition to build model using tabular data, text, and also images.

I teamed up with Rashmi, Abhimanyu, Yiang, Samrat and we finished at 22 among 1917 teams. So far, I have four silver medals and my rank is 542 among 83,588 Kaggler.

This is an interesting competition for me. I was about to quit this competition and Kaggle because of other commitments in life/work. Just one day before team merge deadline, Rashmi asked me to join, at that time, my position is 880-th, about 50%, and Rashmi's team is about 82-th. So I decided to join and finish this competition which I already spent about many hours.

Final Ensemble Models

As part of this team, I worked on final ensemble models. Immediately after join, i completed 5 tasks:

make sure everyone uses the same agreed cross validation schema. This is essential for building ensemble model.
provide model_zoo.md document to keep track of all level 1 models, their train/valid/lb scores, feature used, and file path to their oof/test prediction.
write merge_oof.py to combine all oof/test predictions together.
write R scripts for glmnet ensemble.
write python scripts for LightGBM ensemble.

Once new model is built, other team member update the model_zoo.md and upload the data to a private github repo. Then I update the merge_oof.py to include new models' result, and run glmnet and LightGBM ensemble. We had this ensemble workflow automated so it takes little effort to see the ensemble model's performance.

I spent some times analysing the coefficients/weights of L1 model and tried to exclude models with negative and lower weights. To my surprise it doesn't help at all. The final submission is a glmnet ensemble with 41 models (lgb + xgb + NN).

Also, LightGBM ensemble has much better cv score but the LB score is worse. I suspect it is because there are leakage in L1 models and glmnet is more robust to leakage since it's linear model. Unfortunately, there's no enough time to identify which models have leakage.

Collaboration

This is my 2nd time work in a team, although there's a lot space for improvement collaborating when compared with a professional data scientist team but as night/weekend project, we have done a really good job as a team.

The setup for collaboration:

Slack for discussion. we have channel for general, final_ensemble, random for cat photos etc.
we also used Slack for sharing features which i personal don't like.
Private github repo for sharing code and oof/test predictions.
Monday.com for managing tasks. it gives a nice overview of what everyone's up to.

we tried very hard to get a gold, but other teams work even more harder. At one point we were at 17, and finished at 22.

Some Kagglers to Avoid

Finally, when we waited 1 hour for the final deadline, we had a lovely discussion about our past disqualification experience. We were all shocked when we were at different team in Toxic competition but team up with the same person. We shared their person's multiple Kaggle accounts and added to our personal block-list.

Build Notification Features

2016-07-02T00:00:00+01:00

Data processing time will becomes longer and longer as the increasing rate of data volumes. Users may check-in frequently to see the whether it is finished. Most of the time they will found it hasn't, doing this the users make contact switch which break the flow of whatever the using was doing.

Sometimes the user can't stop doing so, either because they are impatient, or because they really have a deadline to catch. Also, with less likelihood, they might find errors in the processing, either because of the QC check fails, or running out of computational resources.

Giving this fact, it really makes sens to have your program actively inform the user on the process so that they doesn't need to check-in at all. Because users will be notified immediately whenever the whole progressing is completed, or there's error that the user needs to take action onup.

This blog posts walk though the basics of sending Emails in Python, composing and sending out Emails. Each component is broken down into small piceses. It helps you debug/tests Email program, and personalise your Emails. In the end, you should be able to build a email robot.

Prerequisite

Before going into the technical details, you have to check that you are able to send out emails. You need the

[ ] SMTP server,
[ ] user name,
[ ] password,
[ ] port number, and
[ ] communication protocol.

You could easily found out these information from your Email service provider. An example of Gmail is at Here.

To check if you have all the information correct, run the following snippet. It will try to send out an empty Email to yourself. Make sure you fill in the username and password before hit go.

  import smtplib
  username = <Fill In>
  password = <Fill In>
  conn = smtplib.SMTP("smtp.gmail.com")
  conn.starttls()  # set connection to TLS mode
  conn.login(username, password,)  # Log in to the remote server
  conn.sendmail(username, [username], 'For testing')  # Send emails
  conn.quit()  # close connection.

If you don't see error messages, that's great, you are all set. There should be an Email in your Inbox. It has no subject and for testing in the main body.

If you do, you need to double check the information, and try again. If you are sure that the information are correct, btu still can't send out, check your network configuration, maybe the firewall block the connection.

Once you are able to send out an empty email the next is to compose an full Email.

Compose An Email in Python

An Email is consisted of multiple parts, the Subject, Body, attachments, Signature, and also some meta-data including from, to, and date.

Firstly, create an object of MIMEMutiple() class. It will be the building block of your Email. The approached described here is to add each components into it.

Email meta data

Start with meta data.

   msg = MIMEMultipart()
   msg['From'] = 'your_email_address@somewhere.com'
   msg['To'] = 'email_friend_#28473@somewhere.com, email_friend_#122xs2212@somewhere.com'
   msg['Date'] = formatdate(localtime=True)  # standard.

Email body

For plain text, simply add

   body_txt = '''
   Hello,

   Just to tell you Python is awesome.
   '''
   plain_body = MIMEText(body_txt, "plain")

You could also try to compose an complex HTML email in Python, or use diffenret tool to generate the HTML and import in Python.

   html_txt = r'''\
   <html>
   <head></head>
   <body>
   <p>Hi!<br>
   How are you?<br>
   Here is the <a href="http://www.python.org">link</a> you wanted.
   </p>
   </body>
   </html>
   '''
   html_body = email.mime.text.MIMEText(html_txt, 'html')

Attachment

You can attach files of any type in an Email, specially, for image, you could use MIMEImage, and for audio, you could use MIMEAudio.

But you don't have to be specific. MIMEApplication would be sufficeincy for all cases. It will configure the file type of the attached file. Use it as follows:

 with open(fpath, 'rb') as fp:
     part = MIMEApplication(fp.read(), Name=basename(fpath))  # file content as string
     part['Content-Disposition'] = 'attachment; filename="%s"' % basename(f)  # attachment description.
     msg.attach(part)  # attach to the msg.

Put Everything Together

   from os.path import basename
   import smtplib
   from email.mime.application import MIMEApplication
   from email.mime.multipart import MIMEMultipart
   from email.mime.text import MIMEText
   from email.utils import formatdate

   # Configure your email
   username = <Fill in>
   password = <Fill in>
   recipents = <Fill in>
   subject = 'Hey'
   attachments = []  # add attachment here.
   body_txt = '''
      Hello,

      Just to tell you Python is awesome.
      '''

   # Email - meta data
   msg = MIMEMultipart()
   msg['From'] = username
   msg['To'] = ', '.join(recipents)
   msg['Subject'] = subject
   msg['Date'] = formatdate(localtime=True)  # standard.

   # Email - main body
   plain_body = MIMEText(body_txt, "plain")
   msg.attach(plain_body)

   # attachments
   for fpath in attachments or []:
       with open(fpath, 'rb') as fp:
	   part = MIMEApplication(fp.read(), Name=basename(fpath))  # file content as string
	   part['Content-Disposition'] = 'attachment; filename="%s"' % basename(fpath)  # attachment description.
	   msg.attach(part)  # attach to the msg.


   # send out email
   conn = smtplib.SMTP("smtp.gmail.com")
   conn.starttls()  # set connection to TLS mode
   conn.login(username, password)  # Log in to the remote server
   conn.sendmail(username, [username], msg.as_string())  # Send emails
   conn.quit()  # close connection.

Wrap everything in a Class

At JBARML, we are planing to send users emails for a progress update. Many of the data generating process are lured together and automated by a workflow manager. The whole programcan takes upto weeks to complete. Actively sending update progeress to user is much more senssible then user loggin in to a remote machien and check it now and then.

In this case,

email to notify user the progress of the data generating workflow.

etags - Build a TAG for Multiple R Packages

2016-05-04T00:00:00+01:00

Here is what tried to build a TAG for multiple R packages. It enable me to jump to a location where the function/variable is defined and modify if I want to.

Useful variable and functions

ess-r-package-library-path: default path to find packages, should be a list
ess-r-package-root-file: if the folder has DESCRIPTION file, then the folder is a R package.
(ess-build-tags-for-directory DIR TAGFILE): build tag on DIR to TARGET.
tags-table-list: List of file names of tags tables to search.
(visit-tags-table FILE &optional LOCAL): Tell tags commands to use tags table file.

;; new variable 
(defvar ess-r-package-library-tags nil
  "A TAG file for multiple R packages.")

(setq ess-r-package-library-path '("~/tmp/feather/R" "~/tmp/RPostgres/"))
(setq ess-r-package-library-tags "~/tmp/all_tags")

(dolist (pkg-path ess-r-package-library-path)
  (let ((pkg-name (ess-r-package--find-package-name pkg-path)))
    (unless (and pkg-name pkg-path
                 (file-exists-p (expand-file-name ess-r-package-root-file pkg-path)))
      (error "Not a valid package. No '%s' found in `%s'." ess-r-package-root-file pkg-path))
    (ess-build-tags-for-directory pkg-path ess-r-package-library-tags)
    ))

Note the workhorse is ess-build-tags-for-directory which does what it means. The core of this function use find and etags program. The find program will find files with extension .cpp, R, nw etc, and then feed to (using pipe) to the etags program which generate a TAG table. These two steps are demonstrated in the following snippet, which is grabbed from the source code of ess-build-tags-for-directory.

(setq find-cmd (format "find %s -type f -size 1M \\( -regex \".*\\.\\(cpp\\|jl\\|[RsrSch]\\(nw\\)?\\)$\" \\)" (car ess-r-package-library-path)))

(setq regs (delq nil (mapcar (lambda (l)
                               (if (string-match "'" (cadr l))
                                   nil ;; remove for time being
                                 (format "/%s/\\%d/"
                                         (replace-regexp-in-string "/" "\\/" (nth 1 l) t)
                                         (nth 2 l))))
                             imenu-generic-expression)))
(setq tags-cmd (format "etags -o %s --regex='%s' -" "~/lala"
                       (mapconcat 'identity regs "' --regex='")))

(setq sh-cmd (format "%s | %s" find-cmd tags-cmd))
(shell-command sh-cmd)

Note when they are used in Emacs, the tags-table-list variable is appended with the path to the new TAG table. So that the user can use xref-find-definitions (M-.) to jump (if the point is under a word) or select which function/variable to jump to. The users then check the function/variable definition, or modify it if it is necessary. Then call xref-pop-marker-stack (M-,) to jump back.

Compare RPostgres and RPostgreSQL Package

2016-04-14T00:00:00+01:00

R is a great language for R&D. It's fast to write prototypes, and has great visualisation tools. One of constraints of R is it stores the data in system memory. When the data becomes too big to fit in the memory, we asked the user has to manually split the dataset and then aggregate the output later. This process is inefficient and error prone for a non-technical user.

I started an R development project to automate this split-aggregate process. A viable solution is to store the whole data in PostgreSQL, and let R to fetch one small chunk of the data at a time, do the calculation, and then save the output to PostgreSQL. This solution requires frequently data transferring between these two systems, which could be a bottleneck in performance. So I did a comparison of two R packages that interface R and PostgreSQL.

RPosrgreSQL: is supported and developed in the Google Summer of Code 2008 program. It is currently out of development. The last publication is in 2013.
RPostgres: is a new package which provides similar functionality to RPostgreSQL but rewrite using C++ and Rcpp. The development is led by Kirill Müller.

Based on my testing, the RPostgres package is about 30% faster than RPostgreSQL.

The testing set-up is quite simple: I write an R script to send data to and get data out from a remote PostgreSQL database. It logs how long each task takes to complete in R. To avoid other factors that can affect the speed, it repeats this process 20 times and use the minimal run-time as the final score. The dataset transferred between R and PostgreSQL is a flat table with three columns and the number of rows varies from ten thousand to one million.

The run-time in seconds are plotted against number for rows for each package and operation.

Here is a summary of what I observed:

RPostgreSQL is slower than RPostgres. For getting data out, it's 75% slower, which is massive! For writing, difference is closer, it's about 20%. When combine both scores together, it is about 33% slower.
Particularly, it's slower to read than to write for RPostgreSQL package, the ratio is about 1.5. While as it's quicker to read than to write for RPostgres, the ratio is about 0.8. This is an interesting observation.
Both package has a nice feature - the reading/writing time linearly depends on the number of rows. This makes the time estimation reliable. I would be confident to say that for 2 millions rows, it takes RPostgres package about 6 seconds to read.

I don't why which part of implementation makes the RPostgres faster. I guess its the usage of C++ and the magical Rcpp package.

Here is the script just in case you want to your own tests.

library(data.table)                     
library(ggplot2)
library(microbenchmark)
library(RPostgreSQL)
library(DBI)   
                                        # config for PostgreSQL database
host.name <- NULL
database.name <- NULL
postgres.user <- NULL
postgres.passwd <- NULL
postgres.port <- NULL
temporary.table.name <- NULL

                                        # config for testing
nrows <- seq(10 * 1e3, 1 * 1e6, length = 10)
repeats <- 20


                                        # open PostgreSQL connection
pg.RPostgreSQL <- dbConnect(dbDriver("PostgreSQL"),
                           host = host.name,
                           dbname = database.name,
                           user = postgres.user,
                           password = postgres.passwd,
                           port = postgres.port)
pg.RPostgres <- dbConnect(RPostgres::Postgres(),
                         host = host.name,
                         dbname = database.name,
                         user = postgres.user,
                         password = postgres.passwd,
                         port = postgres.port)

ReadWriteWarpper <- function(pg.connection) {
                                        # helper function 
    write <- function() dbWriteTable(pg.connection, temporary.table.name, dt, overwrite = TRUE)
    read <- function() dbReadTable(pg.connection, temporary.table.name)

    var <- list()
    for (n in nrows) {
                                        # create a dataset
        dt <- data.table(x = sample(LETTERS, n, T),  # character
                        y = rnorm(n), # double
                        z = sample.int(n, replace=)) # integer

                                        # read and write once first.
        write()
        read()

                                        # run and log run-time
        res <- microbenchmark(write(),
                             read(),
                             times = repeats)

                                        # parse 
        var[[as.character(n)]] <- data.table(num_row = n,
                                            operation = res$expr,
                                            time = res$time)
    }

                                        # aggregate and return
    rbindlist(var)
}

                                        # run
df0 <- ReadWrite(pg.RPostgres); df1 <- ReadWrite(pg.RPostgreSQL)
df0$pacakge <- "RPostgres"; df1$package <- "RPostgreSQL"
df <- rbind(df0, df1)
plot.df <- df[, min(time) / 1e9, .(num_row, operation, package)]

## generate plot
plot.df[, operation := gsub("\\(|\\)", "", operation)]
ggplot(plot.df, aes(x=num_row, y=V1, col = package)) +
    geom_path() +
    geom_point() +
    facet_wrap(~operation) +
    theme_bw() +
    labs(x="Number of rows",
         y="Run time (sec)"
         )

How to Create a Screencast GIF in Emacs

2015-09-24T00:00:00+01:00

I've always wanted to create a GIF using Emacs to demonstrate some features, it just looks so cool. I finally got a chance after attending the Leeds Code Dojo. The final exercise is bit unusual; we have to write a basic expression evaluation program without using the eval function in whatever language we choose. The first problem we had was to figure out the order of sub-expression to evaluate. For example, in (5 * (2 + 1) ) expression, we know we firstly add 2 to 1 to get the 3, and then multiply 3 by 5. It sounds trivial but it is actually hard to write a program to do that.

I used regular expression¹ to locate the most inner expression to evaluate, then replaced the expression with its evaluating result, and continued these two steps until there was no expression².

The above GIF shows each step in a expression evaluation program written in Emacs Lisp.

This post show how to make GIF in Emacs on Ubuntu system.

Dependencies

There are three packages to install first. We need recordmydesktop to capture the motion of the screen, mplayer to view the video, and imagemagic to convert the recorded video into GIF file. They can be installed easily using the apt-get command, as in the following bash shell script:

sudo apt-get install recordmydesktop mplayer imagemagick

On Emacs side, I use camcorder package to control the workflow. It is hosted in MELPA repository, and can be installed by

(package-install 'camcorder)

Then everything should work nicely together.

Workflow

After these packages are installed, creating a GIF is simply, only requiring three steps.

1. Initiate the recording

In Emacs,

Switch to the buffer we want to record, let's call this buffer the recording buffer,
Initiate the recording by M-x camcorder-record command,
Choose where to save the video file, then

A new frame with the recording buffer will pop up. It is wrapped inside a white rectangular box. Everything inside the box will be recorded and saved in the video file. Note, if we move the window or overlay it with other windows, we probably get undesired results.

2. Record Choose the recording buffer/frame,

Press F-11 to pause/resume,
Show some cool things,
Press F-12 to stop,

Note the demonstration must have an effect on the recording buffer, and we can use with-current-buffer function to dump the output for a particular buffer, for example,

(with-current-buffer "Demo_Buffer"
  (insert "Start to demo: "))

will insert "Start to demo: " into the Demo_Buffer.

I found it is useful to wrap the demonstration into a function and bind to a key because I will probably run it many times.

(defun yt/camcorder-show-off ()
  (interactive)
  (goto-char (point-min))
  (insert "going to show you something cool, don't blink your eyes.")
  (sleep-for 2)
  ;;;; apply some functions
  (insert "\nExciting isn't?"))

(define-key camcorder-mode-map [f5] 'yt/camcorder-show-off)

There are two functions that are helpful control the flow. Use sleep-for function to let the program wait few seconds, and use y-or-n-p to let us choose whether to proceed or switch flow.

3. Make gif

After the demo is finished,

Type M-x camcorder-convert to convert a video file to a GIF file,
Choose a file name for the GIF file,
Select convert method, and choose use mplay with imagicstick.

We probably repeat the step 1-3 multiple times until we are happy with the GIF.

Reference

Recording a GIF screencast of Emacs

Footnotes:

Regular expression might not be suitable for this task, and it works

Everything is actually an expression

Migrate to Ubuntu

2015-09-06T00:00:00+01:00

My MacBookPro's hard drive stooped working last week and I managed to recover most of the data from a Time Machine back-up 6 months ago. But I couldn't get the mu4e and mu working. I feed up with googling, trying, and decide to immigrate to Ubuntu. It would save me from a lot of frustrations and time in making my Mac and office PC work the same way.

Ideally, I will built a Ubuntu on Mac which is exactly the same as the one on my office PC, by just copy over everything ¹. As a minimalist, I decided to build the system from scratch and install software one by one so that I can have an better understanding of what are the necessities for me.

In the last few days, I become extra mindful about the what and how I used the Ubuntu system in the office, and realise the things I need can be grouped into three categories:

Configuration,
1. the .ssh folder for the ssh-agent,
2. the .fonts folder for new fonts,
3. the .mbsynrc file for sync emails,
4. the .ledgerrc.
Software for
1. Development: like git, gcc, Emacs, and R.
2. Writing: org-mode, LaTeX,
3. Email: mu, mu4e, and mbsync.
4. Finance: ledger.
Personal git repositories
1. public reposity on GitHub,
2. private reposities on BitBucket

For 1), since they are small, I can zip up and copy over, or even better, create a git repository so that sync on two machines becomes better easier.

For 2), I need to find the software's package name in the Ubuntu's software repository, and then install all of them by a script. The dependencies should be resolved automatically.

For 3), I need to create a shared folder between the host system and the Ubuntu system, and then copy over the ~/git/ folder.

It really sounds like a plan! I am going to download the Ubuntu installation file now and hopefully the transition will be very smooth.

Footnotes:

see http://askubuntu.com/questions/111236/how-to-migrate-the-whole-system-to-a-new-machine

My Expeirence with Repetitive Strain Injury (RSI)

2015-07-19T00:00:00+01:00

Someday I typed more than 80 thousand times just in Emacs. This is pretty awesome at first sight but it can cause serious health problem.

Last month, I felt burning pain of my forearms. It is an symptom of Repetitive strain injury (RSI). I realised that if continue typing like that, one day I will never able to do programming, like the Emacs celebrities in Xah Lee' article about RSI.

Since then I've deliberately tried to avoid aimless and unproductive typing, take more typing breaks, think though things before trying, write more on paper.

Conditions are getting better: I don't feel server pain any more, only sometimes uncomfortable.

But I need to find a better way to improve it. Because sometimes I got the idea, but can't touch the keyboard. This feeling really suck.

So I investigated the Hydra package and use it to group related commands together so that use only two keys are needed to perform frequent tasks.

For example, to search something in current project, instead of typing M-x helm proj grep, that's 16 keystrokes, I only need F5 G with Hydra. The implementation is listed in this post.

But calling functions/commands in Emacs counts only a small proportion of my typing; most of the time, I write code and report.

This is where Yasnippets kicks in, it enable me to type less without losing quality. For example, I use this snippet quite often when writing R code,

res <- sapply(seq_len(n), function(i) {
    ## 
})

That's more than 40 keystrokes. Yasnippets can short it to only 6s! After I type sapply and then hit TAB, it will expand to the region above.

I will investigate the Yasnippet package next week. If you know any good tutorials for Yasnippet or snippets for writing R code, please share your resources.

Start Enjoying Regular Expression In Emacs

2015-06-30T00:00:00+01:00

The search-forward-regexp, replace-match, and match-string functions work together nicely, and makes my job much easier and enjoyable!

I am writing a release notes for the a software updates. Part of the process is to associate the SVN Revision number that relates to important changes, so that others can backtrack and review the code and see what exactly has been implemented.

In Phabricator, the revision number will be render automatically. Clicking them takes me to the exact revision, showing the difference with previous version. But the documentation will be eventually built by Sphinx and hosted on a remote server. So I have to manually add the URL to all the SVN revision number. For example, to replace rS1234 to

[[http://phabricator.domain.co.uk/rS1234][rS1234]]

There are 31 revision number in the whole document. I could do it manually but for the long term benefits, it would be more efficient write a function to process it automatically, maybe others can use it as well.

Implementation

The first thing I noticed is each SVN revision numbers consist of two letters (rS) and few digits. Because the four digits I don't know beforehand, I have to use regular expression to do the pattern search.

The tricky bit here is to retrieve the values that matched the pattern, because of it is needed to construct the URL that points to the commits, and I also need to replace the it with differnet values.

The procedure can be summarised as:

Find the revision number that match the patterns described above. I use search-forward-regexp() to search the pattern "rS[0-9]+", which means a string that starts with rS with one or more digits.a
retrieve the values that matched the pattern. This is done by match-string().
replace the revision number with the constructed URL. This is done by replace-match(), and I use concat() to combine the IP address with the revision number.

The following is a workable implementation:

(defvar revision-pattern "rS[0-9]+"
  "The RegExp pattern of the SVN revision number")

(defvar repo-url "http://10.0.0.11/"
  "The IP address of the SVN repository")

(defun yt/add-link-to-SVN-revision-number ()
  "add links to svn commits identifier"
  (interactive)
  (while (search-forward-regexp revision-pattern)
    (let* ((commit (match-string 0))
           (link (concat repo-url commit)))
      (replace-match "")
      (org-insert-link nil link commit))))

Note the last two lines of the function can be simplified as

(replace-match (concat "[[" link
                       "][" commit "]]"))

You can easily adopt the code and make it applicable to your case, just modify the revision-pattern and repo-url variables. But beware that you should not apply the function to the same buffer more than once, otherwise you will get something crazy like this:

[[http://10.0.0.11/[[http://10.0.0.11/rS1234][rS1234]]][[[http://10.0.0.11/rS1234][rS1234]]]]

One way to make it better is to have a test before replacing: if the revision number is already associated with a URL, then do nothing. If you have figure out how to do it, please let me know and I've happy to update this post.

My posts published last year showed my frustration with regular expression in Emacs. But now I am looking forward doing more text processing with it, because it will be fun!

The search-forward-regexp, replace-match, and match-string functions work together nicely and make the my job much easier and enjoyable!

What's your favourite functions in regular expression? Do you have something to recommend?

Import Irregular Data Files Into R With Regular Expression - an BODC Example

2015-06-25T00:00:00+01:00

The Irregular Data Files
First Attempt - Skip Lines
Second Attempt - Remove Lines
Third Attempt - Capture Lines (RegExp)
Code and Sample Data

The first step in data analysis is to get the data into the modelling platform. But it may not be as straightforward as it used to be since nowadays statistician are more likely face the data files that are not in CSV or others format that can feed directly to the read.table() function in R, in which cases, we need to understand the data files in terms of the structure and apply pre-process first. My general strategy is to discard the unnecessary information in the data files and hopefully leave a regular data files.

In my last week's post, Why I Should Explore Regular Expression and Why I Haven't, I expressed my interests in Regular Expression and lucky I got a chance to use it for getting the data into R. It provides me a different strategy: pick only what I am interested in.

The Irregular Data Files

The task is simple: I have about 1,800 .text data files downloaded from British Oceanographic Data Centre (BODC). They are the historical tidal data and are separated by year and by port. I need to combine all the data into one giant table in R, and save it later for modelling.

One sample data file looks like this:

Port:              P035
Site:              Wick
Latitude:          58.44097
Longitude:         -3.08631
Start Date:        01JAN1985-00.00.00
End Date:          03OCT1985-19.00.00
Contributor:       National Oceanography Centre, Liverpool
Datum information: The data refer to Admiralty Chart Datum (ACD)
Parameter code:    ASLVZZ01 = Surface elevation (unspecified datum) of the water body                      
  Cycle    Date      Time      ASLVZZ01     Residual  
 Number yyyy mm dd hh mi ssf           f            f 
     1) 1985/01/01 00:00:00      1.0300      -0.3845  
     2) 1985/01/01 01:00:00      1.0400      -0.3884  
     3) 1985/01/01 02:00:00      1.2000      -0.3666

The first 9 lines are the metadata, which describes the port ID, name and location of the port, and other information about the data. The line 10 and 11 are the headers of the data matrix.

First Attempt - Skip Lines

After the glimpse of the data sample, my first thought was to skip the first 12 lines and treat the rest as a regular data files that has space as separator. It can be easily done by using read.table() with skip = 12 option.

read.table(data.file, skip = 12) ## error

It turned out this approach won't work for some files because when the way of measuring tidal were changed, the date and port were highlighted, leaving a second chunk of data matrix but again with metadata and few other characters. It looks like this:

;; end of first chunk 

########################################
 Difference in instrument
########################################

Port: P035
;; other metadata

Second Attempt - Remove Lines

Although the first attempt isn't success, I've learnt a bit about the structure of the data files. And based on that, I came up with a second approach: read the data files into R as a vector of string, one element for a line, and then remove all the lines which are metadata. They start with Port:, Site: or Longitude: etc or the ### chunk. It can be done using grep function, which tells me exactly which element of the vector contains the metadata.

s <- readLines(data.file)
metainfo.list <- c("Port:", "Site:", "Latitude:", "Longitude:", "Start Date:", "End Date:", "Contributor:", "Datum information:", "Parameter code:")
meta.line.num <- sapply(metainfo.list, function(i) {
    grep(pattern = i, s)
})
res.2 <- s[-meta.line.num]

This approach works well as long as the metainfo.list contains all the lines I'd like to remove. The downside is that I won't able to know I've includes all of them until the whole process is finished. So when I was waiting for the program to finish, I came up with a third approach, a better one.

Third Attempt - Capture Lines (RegExp)

The above two approaches are to discard the unnecessary information, but I may be in the situation that there are other lines that should be discard but I haven't encounter yet, then the process becomes tedious try-error and takes quite long.

Equally, another approach is to select exactly what I am interested in by using regular expression. But first, I have to identify pattern. Each data point was recorded at a certain point, and therefore must be associated with a timestamp, for example, the first data point is recorded at 1926-01-01 00:00:00. They also has an ID values with an closing parentage's, for example 1.

1) 1985/01/01 00:00:00      1.0300      -0.3845

So the content of my interests are have a common pattern that can be summarised as: the lines that start with a number of spaces, and also have

observation ID: few integers, and an ending parentheses,
observation date: few integers with forward slashes that means year, month and day, and then a space,
observation time: few integers with colons, means hour, minutes and seconds.

The patterns in RegExp can be formulated as the roi.pattern variable and the whole process can be implemented as:

roi.pattern <- "[[:space:]]+[[:digit:]]+\\) [[:digit:]]{4}/[[:digit:]]{2}/[[:digit:]]{2}"
roi.line.num <- grep(pattern = roi.pattern, s)
res.3 <- s[roi.line.num]

To me, there isn't an absolute winner between the second and third approach, but I prefer to use regular expression because it has more fun with it; I am a statistician and like to spot patterns.

Also, it is an direct approach and more flexible. Note I can continue to add components to the regular expression to increase the confidence in selecting the right data matrix. For example, there are spaces and then few integers at the timestamp. But it will presumably increase the run-time.

Code and Sample Data

You can download the exmaple data and run the scripts listed below in R to reproduce all the results.

#### * Path
data.file <- "~/Downloads/1985WIC.txt" ## to the downloaded data file

#### * Approach 1
read.table(data.file, skip = 11) ## error

#### * Approach 2
s <- readLines(data.file)
metainfo.list <- c("Port:", "Site:", "Latitude:", "Longitude:", "Start Date:", "End Date:", "Contributor:", "Datum information:", "Parameter code:")
meta.line.num <- sapply(metainfo.list, function(i) {
    grep(pattern = i, s)
})
res.2 <- s[-meta.line.num]

#### * Approach 3
roi.pattern <- "[[:space:]]+[[:digit:]]+\\) [[:digit:]]{4}/[[:digit:]]{2}/[[:digit:]]{2}"
roi.line.num <- grep(pattern = roi.pattern, s)
res.3 <- s[roi.line.num]

Why I Should Explore Regular Expression and Why I Haven't

2015-06-18T00:00:00+01:00

Like many R users who are not actually programmer, I am afraid of regular expression (RegExp), whenever I saw something like

I'd told myself I won't be able to understand it and gave up on the sight.

But I've collected few RegExp patterns that do magical jobs. My favourites are the dot (.) and dollar ($) sign and I usually use them with list.files() to filter the file names in a directory. For example,

list.files(pattern = ".RData$")
list.files(pattern = ".text$")

The first line returns all the R image files, which have file names ending with RData, and for the second all the text files which have file names ended with text. Basically in regular expression, dot sign (.) means anything, and dollar sign ($) means the end of a string. By combining these two, I am able to select multiple files with certain patterns, without manually picking one by one.

How powerful is that! It is an inspirational example that motivates myself from time to time to look deeper and get my head on the topic of regular expression. But I just couldn't have a clear picture of how to us it.

I think the main problems for me to understand RegExp in R are

The syntax is content-sensitive

A subtle change can lead to random results. For example, the above pattern can also be \\.RData$, which means file names ended with .RData. The dot (.) sign here literally means ".". Adding two backslashes \\ changes the meaning of the pattern completely, but both gives the same results. It gave me so much frustration when extrapolating a pattern that works in one case to a similar case but get random results.

The syntax is hard to read

The RegExp pattern above are reasonably easy to understand, if one spent 10 minutes reading the manual, but the following is just crazy.

m <- regexec(pattern = "^(([^:]+)://)?([^:/]+)(:([0-9]+))?(/.*)", x)

There are 12 parentheses, 6 square brackets and many other symbols. Even same symbol have different meanings, and it's hard to find out exactly what they means because

There isn't enough learning materials

I've never seen an R book that mentioned regular expression. This topic is certainly not a teaching content in university courses or training workshops.

Even google fails to find any meaningful resource except for the Text Processing in Wiki, which is the best I could find.

Although there are related questions in StackOverflow, most of the answers were set in a very specific situation. It's hard make it applicable to other situations or learn this topic from the discrete Q&As.

It has created a mental barrier that statistician shouldn't teach nor learn RegExp at all, or at least for me. But my limited experience suggests that it is such a powerful feature that I've missed a lot.

But

I believe there will be more chances to process text files, for example, parse the log files of this blog. RegExp can improve the efficiency to a great extent. So I am considering to invest the time to learn it properly.

Are you a R user? What's your experience with regular expression? Do you have good learning materials to recommend? If so, please share your experience on the less-talked area.

Use Emacs's Org-mode to Effectively Manage Small Projects

2015-06-14T00:00:00+01:00

Organising
Managing
Monitor

DEADLINE: <2015-06-09 Tue 20:00>

Org-mode is great to serve as knowledge management tool, it also has helped me increase my personal effectiveness. Recently I have been exploring org-mode for managing small projects in the business environment, in which collaboration happens occeasionally between me and the project team members.

In this post I summarised my workflow to organise, manage and monitor a project. The implementation of this workflow revolves around the collaboration. I have been practise this workflow for a while and can see my growth in planing and managing skills.

Organising

I use a broad definition of project: as long as a task that requires a series of sub-tasks to be done, then it is a project. Normally I categories any tasks that relates to a project into three groups:

Project Tasks: the major tasks that must to been done in order to deliver the project product.
Tasks: administrative or miscellaneous tasks that keep the project goes on, like sent out the invoice.
Notes: anything that is important to the project and therefore worthy keeping a record, like meeting notes or decision made that that impacts the project progress.

Each category has a corresponding top level section or heading. Once this outline is setup, it is very convenient to view content under these categories, regardless of what tasks I was working on, either reading emails, coding, or writing report. Org-mode can scan all the .org files in a direcotry, and creates a tree-structure, with the file name being the root, and headings being the nodes.

An intuitive way to locate a any node is to start from the beginning, the process is same as finding a section in a text book. It can be summarised as:

first, find the right book by its name,
then find the right part,
then narrow down to the right section,

and continue to the section I am interested in. An more pleasure way is to use fuzzy match supported by Helm package - I can narrow down the selection by random nodes. For example, as the images below shows, to locate headline under this article among 40 org files, I only need to search "Small pro", because there are only three headlines has "Small" in its name, "small changes", "small talk", and "small project", and "pro" narrow down to the unique headline.

It saves me a lot of time in remembering where I saved one notes, and wandering around the files to find something. I only explain a bit of the features of Helm, if you want to try out, you can find my configuration here. I recommend a good tutorial if you want to know more.

Figure 1: Test image

We usually a couple of projects at the same time. Also, create a new tasks or notes is easy. org-capture-mode would create a temporary node and by default it will be saved as a subtree in refile.org, or I can directly re-locate the headline directly to this project using the locating mechanism above.

These two features are most enjoyable to use, and make me away from wandering in multiple directories, trying to find the right files, and therefore increase my productivity. Never under estimate how long you will spent in finding in one file.

Managing

Projects usually come with hard deadlines about the product delivery. Setting and change deadlines in org-mode is pleasurable with org-deadline C-c C-d.

It brings up a mini-calendar buffer (shown below), I can use shift+left and shfit+right to move forward and backward for a day, or shift-up and shift-down to move between weeks, and hit RET to select a deadline. Apart from navigating, I can also choose to type the exact date directly, like "2015-07-25" and hit RET.

Once the deadline is set it will show up in that day's calendar. I don't want to suddenly realise there is a deadline I had on that day. So it makes sense to have an early warning period to show the tasks if it is due in days. This behaviour is governed by the org-deadline-warning-days variable. In my Emacs configuration, I set to 30 days. It gives me plenty of time to do any tasks.

I also set deadlines for sub-tasks since it is quite easy to do in org-mode. But coming up with realistic deadlines is difficult. To me, it must give enough time to do the task properly, to the PM, it must be fit in the whole project plan and resource. Both are likely to have different opinion on how long to implement the new features with documentation. It is quite important skills to have: to me, it reflects my understand on the problem and also my own technical capability, to the manager, it is part of their project plan.

My initial estimation may be far from the actual effort, especially when the problem domain is new to me, or I haven't done similar tasks before. The more I do, the better I am good at estimating. At this stage, I practise this skill seriously, and like to have someone with more experienced to review my estimation.

To make this task easy for them, I'd present an overall view of the project time-lines, which clearly shows the period allocate to the specific tasks. org-timeline will generate a time-sorted view for all the tasks. The recent feedback I received is that I tend to overlook the time spent on documentation and tests. Someone with more than 10 years in software development says they usually takes about 3x times on these two tasks together than actually coding.

time-line view also provides benchmark to the progress and I check it frequently to make sure I am on track. It gives the PM a reference for swapping tasks if some becomes urgent.

Monitor

Additional to have the early warning system to prevent sudden surprise, org-mode provides another way of monitoring the project in terms of resource - the actual time I spent on the project. This feature is quite useful when I am given a quite loose deadline but with limited resource, say 150 hours.

Since the sub-tasks are mostly defined in the early stage, whenever I start to do it, I clock in first by org-clock-in. The clocking will be stopped once I manually clock out, or clock in to another task, or the tasks is completed (marked as DONE.) For each clock entry, it shows start time, end time and duration.

Multiple clocking logs are accumulated, and each entry shows the start time, end time, and duration. The durations can be added up and tells me exactly how much time I spent on each tasks. The whole tasks under the project and aggregated across the whole project, by one single function org-clock-report (C-c C- C-r).

Table 1: Clock summary at [2015-06-14 Sun 11:17]
Headline	Time			Effort
Total time	10:41
TODO Use Emacs's org-mode to Manage a Small Project	10:41
TODO Tasks		1:45
DONE add example for org-refile			0:35	0:30
NEXT add example for org-clock-report			0:13	0:15
NEXT proof read			0:11	0:15
NEXT proof read - 2			0:46	1:00

It is normal to underestimate the complexity of an tasks, and spent too much time in resolve them, and usually I can catch up the in the later stage, however if I had the feeling the overall progress has been affected, I need require more sources from the PM, and the quote I will give is extra hours I had based on my initial estimation. That's an quick reaction.

Also, the clock-report table tells me the different between my effort estimation and the actual time I spent on that tasks.

Control the Plotting Order in ggplot2

2015-05-06T00:00:00+01:00

The above two plots show the same data (included below), and if you are going to present one to summarise your findings, which will you choose? It is very likely you are going to pick the right one, because

the linear increasing feature of bars is pleasant to see,
it is easier to compare the categories, the ones on the right has higher value than the ones on the left, and
categories with lowest and highest value are clearly shown,

In this article I am trying to explain how to specify the plotting orders in ggplot to whatever you want and encourage R starters to use ggplot2.

To create a bar plot is dead easy in R, take this dataset as an example,

mode	count
ssh-mode	2361
fundamental-mode	4626
git-commit-mode	4869
mu4e-compose-mode	4964
emacs-lisp-mode	6205
shell-mode	10046
minibuffer-inactive-mode	12624
inferior-ess-mode	25774
ess-mode	47115
org-mode	78195

to get the plot on the right side, reorder the table by count (it is already been done), then

with(df, barplot(count, names.arg = mode))

will do the job. That's simple and easy, it does what you provide. This is completely different to ggplot() paradigm, which does a lot computation behind the scene.

ggplot(df, aes(mode, count)) + geom_bar()

will give you the first plot; the categories are in alphabetically order. In order to get a pleasant increasing order that depends on the count or any other variable, or even manually specified order, you have to explicitly change the level of factors.

df$mode.ordered <- factor(df$mode, levels = df$mode)

create another variable mode.oredered which looks the same as mode, except for the underlying levels are in different. It is set to the order of counts. Run the same ggplot code again will give you the plot on the right. How does it work?

First, every factor in R is mapped into an integer, and the default mapping algorithm is

sort the factor vector alphabetically,
map the first factor to 1, and last to 10.

So emacs-lisp-mode is mapped to 1 and ssh-mode is mapped to 10.

What the reorder script can do is to sort the factors by count, so that ssh-mode is mapped to 1 and org-mode is mapped to 10, I.e. the factor order which are set to the order of count.

How does this affects ggplot? I presume ggplot do the plotting on the order of levels, or let's say on the integer space, I.e. do the plotting from 1 to 10, and then add the labels for each.

In this example, the default barplot function did the job. Usually we need to do extra data manipulation so that ggplot will do what we want, in exchange for the plot good better and may fits in the other plots. Without considering the time constraints, I would encourage people to stick with ggplot because like many other things in life, once you understand, it becomes easier to do. For example, it is actually very easy to specify the order manually with only two steps:

first, sort the whole data.frame to a variable,
then change the levels options in factor() to what ever you want.

To show a decreasing trends - the reverse order of increasing, just use levels = rev(mode). How neat!

Group Emacs Search Functions using Hydra

2015-04-17T00:00:00+01:00

I am a search-guy: when I want to know something, I use the search functionality to locate to where has the keyword, and I didn't use my eyes to scan the page, it's too slow and harmful.

Emacs provides powerful functionality to do searching. For example, I use these commands very often (with the key-binds),

isearch (C-s), search for a string and move the cursor to there,
helm-swoop (C-F1), find all the occurrences of a string, pull out the lines containing the string to another buffer where I can edit and save,
helm-multi-swoop M-X, apply helm-swoop to multiple buffers, very handy if I want to know where a function is called in different buffers.
projectile-grep or helm-projectile-grep C p s g, find which files in current project contains a specific string, similar to helm-multi-swoop limits the search to files in project directory.

I love doing searching in Emacs, but the problem is to have to remember all the key-binds for different tasks. Also, sometimes, I forgot about what alternatives I have and usually go with the one that I most familiar with, which usually means not the right one. I sometimes realise I use isearch multiple times to do what ace-jump-word-mode can achieve by just once.

Org-mode Hydras incoming! gives me some idea to group all these functions together, and press a single key to perform different tasks, so this can free my mind from remembering all the key-binds. Also, I can write the few lines of text to reminds myself when to do what, and this potentially can solve problem two.

Here is the hydra implementation for searching:

(defhydra hydra-search (:color blue
                               :hint nil)
  "
Current Buffer : _i_search helm-_s_woop _a_ce-jump-word
Multiple Buffers : helm-multi-_S_woop
Project Directory: projectile-_g_rep helm-projectile-_G_rep
"
  ("i" isearch-forward)
  ("s" helm-swoop)
  ("a" ace-jump-word-mode)
  ("S" helm-multi-swoop)
  ("g" projectile-grep)
  ("G" helm-projectile-grep))
(global-set-key [f4] 'hydra-search/body)

So next time, when I want to search something, I just press F4, and then it brings up all the choices I have, and I don't need to worry about the key-binds or which to use! That's cool!

I am looking forward simplifying my Emacs workflow using hydra package, the key challenge is to identify the logical similarities among the tasks and then group them together accordingly. For hydra-search(), it is "search something on somewhere".

A Workflow for Using Git to Track SVN Repository

2015-04-16T00:00:00+01:00

Version control system is a complex issues, and hard to understand the idea of branching and different types of merging. I merely understand the basic of Git, and it already makes my life a lot easier, I am managing about 10 repositories at this moment without much effort.

But my collages are using SVN as the centre storage for scripts. Switching to SVN is not a problem, I just need few weeks to transfer the knowledge and start to use it. I am reluctant to learn something basic and have duplicated knowledge, also, I use GitHub and Bitbucket which are Git based. But sticking to Git make mine work impossible to work with collauges.

Then I found out the Git developer has already made effort to bridge Git and other version control system, like SVN. The git svn allows me to just Git commands for staging, cherry-picking, pull etc, and then upload to the SVN remote repository with just one command line. I really like the idea of transferring the skills from one system to another without any cost, it makes me believe Git is great and I can continue to use Magit in Emacs!

Here is the basic steps and comments for this work flow:

Create a folder mkdir ProjRepo
Create an empty Git repository git init
Add the following to .git/config

[svn-remote "svn"] url = https://your.svn.repo fetch = :refs/remotes/git-svn

and change the URL to right repository,

pull from SVN central repository to this folder, git svn fetch svn
switch to SVN remote branch, git checkout -b svn git-svn
modify or add files
use git add and git commit for snapshot local changes
sometimes need to update local repository, git svn rebase
finally upload local changes to SVN central repository git svn dcommit

See the official manual 8.1 Git and Other Systems - Git and Subversion git-svn documentation for more details.

Why Use Emacs 1 - Emacs Speaks Statistics

2015-01-28T00:00:00+00:00

I am a Statistician, coding in R and write report is what I do most of the day. I have been though a long way of searching the perfect editor for me, tried Rstudio, SublimeText, TextMate and settled down happily with ESS/Emacs, for both coding and writing.

There three features that have me made the decision:

Auto Formatting

Scientists has reputation of being bad programmers, who wrote code that is unreadable and therefore incomprehensible to others. I have intention to become top level programmer and followed a style guide strictly. It means I have to spent sometime in adding and removing space in the code.

To my surprise, Emacs will do it for me automatically, just by hitting the TAB and it also indents smartly, which make me conformable to write long function call and split it into multiple lines. Here's an example. Also, if I miss placed a ')' or ']' the formatting will become strange and it reminders me to check.

rainfall.subset london,
rainfall.pairs,
rainfall.dublin)

Search Command History

I frequently search the command history. Imaging I was produce a plot and I realised there was something miss in the data, so I go back and fix the data first, then run the ggplot command again, I press Up/Down bottom many times, or just search once/two times. M-x ggplot( will give me the most recent command I typed containing the keyword ggplot(, then I press RET to select the command, which might be ggplot(gg.df, aes(lon, lat, col = city)) + geom_line() + ...... If it is not I want, I press C-r again to choose the second most recent one and repeat until I find right one.

Literate Programming

I am a supporter of literate statistical analysis and believe we should put code, results and discoveries together in developing models. Rstudio provides an easy to use tool for this purpose, but it does not support different R sessions, so if I need to generate a report, I have to re-run all the code from beginning, which isn't particle for me with volumes data because it will take quit long.

ESS and org-mode works really well via Babel, which is more friendly to use. I can choose to run only part of the code and have the output being inserted automatically, no need to copy/paste. Also, I can choose where to execute the code, on my local machine or the remote server, or both at the same time.

These are only the surface of ESS and there are lot more useful features like spell checking for comments and documentation templates, that makes me productive and I would recommend anyone uses R to learn ESS/Emacs. The following is my current setting.

;; Adapted with one minor change from Felipe Salazar at
;; http://www.emacswiki.org/emacs/EmacsSpeaksStatistics
(require 'ess-site)
(setq ess-ask-for-ess-directory nil) ;; start R on default folder
(setq ess-local-process-name "R")
(setq ansi-color-for-comint-mode 'filter) ;;
(setq comint-scroll-to-bottom-on-input t)
(setq comint-scroll-to-bottom-on-output t)
(setq comint-move-point-for-output t)
(setq ess-eval-visibly-p 'nowait) ;; no waiting while ess evalating
(defun my-ess-start-R ()
(interactive)
(if (not (member "*R*" (mapcar (function buffer-name) (buffer-list))))
(progn
(delete-other-windows)
(setq w1 (selected-window))
(setq w1name (buffer-name))
(setq w2 (split-window w1 nil t))
(R)
(set-window-buffer w2 "*R*")
(set-window-buffer w1 w1name))))
(defun my-ess-eval ()
(interactive)
(my-ess-start-R)
(if (and transient-mark-mode mark-active)
(call-interactively 'ess-eval-region)
(call-interactively 'ess-eval-line-and-step)))
(add-hook 'ess-mode-hook
'(lambda()
(local-set-key [(shift return)] 'my-ess-eval)))
(add-hook 'inferior-ess-mode-hook
'(lambda()
(local-set-key [C-up] 'comint-previous-input)
(local-set-key [C-down] 'comint-next-input)))
(add-hook 'ess-mode-hook
(lambda ()
(flyspell-prog-mode)
(run-hooks 'prog-mode-hook)
;; (prog-mode)
))

;; REF: http://stackoverflow.com/questions/2901198/useful-keyboard-shortcuts-and-tips-for-ess-r
;; Control and up/down arrow keys to search history with matching what you've already typed:
(define-key comint-mode-map [C-up] 'comint-previous-matching-input-from-input)
(define-key comint-mode-map [C-down] 'comint-next-matching-input-from-input)

Send Stylish MIME in Emacs

2015-01-15T00:00:00+00:00

Last Updated: 18 Jan 2015

This is the first technical article in this blog, however the main purpose is not to analyse the problem and provide the solutions, but to tell a story of an ordinary person trying to pursuit his vision in a multi-languages environment (Emacs and HTML) that he only knows the basis. Hope you find it is interesting to read and for those who care the solution more than problem-solving approach, please see the last section.

The Problem
HTML Attachment Solution
Paradise of MIME
Hack org-mime
MIME Solution
Emacs Configuration

The Problem

The first time I thought I need an fancy Email is when I sent an quick model update to my colleague; I have a table like this

Conditioning Variable	Dependent Variable	Probability
k >= 50	t >= 50	0.154
k >= 50	t >= 100	0.111
k >= 50	t >= 200	0.078

It was written in org-mode in which I can do the formatting quickly and nicely. But once copied over to Outlook, it looks messy, and the columns does not lineup.

| Conditioning Variable | Dependent Variable | Probability |
|-----------------------~~--------------------~~--------–—|
| k >= 50 | t >= 50 | 0.154 |
| k >= 50 | t >= 100 | 0.111 |
| k >= 50 | t >= 200 | 0.078 |

The correct way is to insert a table in Outlook. First, I have to export the table to a CSV file, than open it in Excel, and finally copy it over to Outlook which will recognised it as a table.

HTML Attachment Solution

I guess the purpose of that email is to give my colleague few numbers, in a way that he can compare and gain a feeling of the model. So the format is really necessary, but the workaround is really tedious.

I have another colleague who is an HTML expert and produced an company CSS style-sheet. He was kindly customised it to match the org-export class, i.e. org-ur, org-table, org-list.

So what I did was to export to org-file as a HTML and attached it in the email so that my colleague can simply click and open it in a browser, which will gives him a nicely formatted table. But people have hundreds email per day and seems to dislike attachments.

Paradise of MIME

I noticed Bernt Hansen pointed out in his famous Org Mode - Organize Your Life In Plain Text! that he use org-mime to sent HTML Email. MIME, standards for Multi-Purpose Internet Mail Extensions, is an extension to plain email and enable user to exchange rich data includes image, table, video etc.

The org-mime can parse the org file into an HTML code, in a way that the email server like Office365 or Gmail can recognise and render it with pre-defined styles.

The default style looks awful: the font, the colour, size, basically nothing is right. Recently I sent about 3-5 emails using this style, I doubt the reader will spent less time in reading and comprehend it, therefore the message is not conveyed.

But the workflow is fascinating: I call org-mime-subtree function, then I just type few email address, no need to switch to system or Outlook, everything is done in Emacs and at the exact point where the main content is generated.

So I was thinking, what if the email is look as good as the attachment? What if I can apply the style to the email, that would be looks fanatic!

I did my research, the org-mime indeed provides feature to let user to change the HTML style, two example are showed on worg. The package first generate the HTML file, and than search-and-replace a certain chunk, for example,

1
2
3
<p> 
  this is a paragraph 
</p>

will becomes something like this, depends on users specification,

1
2
3
<p style="blue">
  this is a paragraph 
</p>

The search-replace mechanics works fine, for a small email. It takes a pair value (element, style), where element can be paragraph, table, list and style can be colour, font, size etc. The problem is this pair is not quick match the standard CSS file,

1
2
3
4
5
6
7
body {
    font-family: "Helvetica Neue", "Lucida Grande", "Lucida Sans Unicode", Helvetica, Arial, sans-serif !important;
    font-size: 14px;
}
body #content {
    padding-top: 70px;
}

One can processing the CSS file, and feed the package a long list of pairs. But this approach seems not safe. I quickly skim the CSS file and found something I couldn't understand, for example the body #content block above.

Hack org-mime

I think the most problem-free approach is to follow org-export-html and ensure the generated Email has same style as exported HTML and org-MIME package will eventually implement this, but I don't to wait and decide to hack.

The script is formatted in a nice way, and looks like a textbook C program: it first declares variable and functions, with concise documentation so one can visualise the structure after reading 5-10 minutes. But the implementation is way beyond my knowldge on Emacs-Lisp language. I almost looked up each function that be used, take this snippet for example,

1
2
3
4
5
6
 
(with-temp-buffer
  (insert html)
  (goto-char (point-min))
  (run-hooks 'org-mime-html-hook)
  (buffer-string))

I have no idea of what does it means. You know the feeling when you try to learn an foreign language but took the wrong book that way above your level, and you find there no single word you could understand, and you was like What The Hell? That was my feeling.

The strategy I came up was to build up my Emacs-Lisp vocabulary: try to understand the functions/processes and translate it into a plain English, for example,

with-temp-buffer: create a temporary buffer
insert: insert the string, in this case, called html, at point.
goto-char: move the cursor, which is called point in emacs, to somewhere
point-min: means the begining of a buffer/file
run-hook: run functions that links to org-mime-html-hooks
buffer-string: return a buffer as a string

Now that I understand each words, I need to comprehense it and combine than together to understand the mean of this snippet. I tried to write in a plain English and the first attempt is like this

create temporary buffer, insert the generated html file, than move the cursor to the very start, and than apply other functions that links to org-mime-htmize

I continue this word-sentence-paragraph process and I understand few functions. But it can goes on and on, and the more I learn about Emacs lisp, the future away I digress from my original goal: apply the style sheet to HTML email. I guess this is a common dilemma in working with multi-languages. Usually I follow my interests but this time I choose to focus on achieving the goal.

MIME Solution

It turned out it is a right decision. The concept of "inline-CSS" is mentioned int the script, I googled and found out the solution within 10 minutes. I realised that what I need to do is add a block in beginning of the HTML mail!! BINGO!

1
2
3
4
5
6
<head>
  <style>
    ...
  </style>
</head>
;; html email content starts here

Emacs Configuration

Here's the settings:

1
2
3
4
5
6
7
8
9
10
11
12
(require 'org-mime)
(add-hook 'org-mime-html-hook
          (lambda ()
            (insert
             "       
<head>
<style>
;; content of the .css file 
</style>
</head>"
             ))
          t)

Emacs for Writing

2014-12-26T00:00:00+00:00

Last Updated: 31 Dec 2014

Do you use Emacs for writing the LaTeX, Markdown, or org documents? Do you have a set of specific settings only for writing? In this article I will share my experience of configuring a writing mode in Emacs that make it the most efficient writing tool for me.

Word Count
Variable-width Font
Sentence Highlight
Wrap-up

Word Count

I try to write as concise as possible and I use word count as a benchmark. Counting the words does not sounds like a trivial task in my cases because I have a habit to comment, even for general writing. I may comment out the whole paragraph, and leave a note aside about why, which are kept as it will be helpful in edit/review. These comments and notes should not be counted since the reader can't see them.

Addition to comments, there is a full list that does not count for technical articles, like source code, tables, figure captions etc. Some people may add reference section to the list as well.

org-wc provides the org-wc-subtree function that know what to count and what not to count. Also, org-wc-display will loop though all sections and overlay the number of words to each section headline. It is particularly useful when I need to know which sections needs to trim down and which to add more.

One of my daily achievement is to complete a writing challenge, which is about either to have write about 500 words or 45 minutes, whichever comes first. It is like a racing game for me, knowing the time or number of words is important. Tracking time is simple in Org-mode but words is problematic: I have to call the org-wc-subtree function manually. I raised a issues on GitHub and guided to nanowrimo mode, which updates the word counts while I am typing and shows it on mode-line.

It works out of box for me. The number of words is adjacent to the time I spent, which make it is very convenient to compare. Also, it calculate the average number of words per minute. It use this number to predict how long I need to achieve my daily goals (which is 500 words). The picture above shows that I spent 30 minutes editing and there are 254 words in this section.

Variable-width Font

I have a little OCD about font since university. I use Time News Rome for formal report and any other serif font for general writing because they make paragraph and text easier to read.

There was a time my friend passed me a PDF file and asked me to review it. The problem was it was in Arif font (I think) which looks terrible, and also writing became unpleasant. This experience makes me to think what is the best font for writing.

I did some research and come across the concept of variable-width font. As a programmer, I use Adobe's Source Code Pro font as default which means I face monospaced font all day. For a monospaced font, each character has same space.

While for variable-doth font, each cahracter takes width corresponding to it's shape. For example, the length of "i" is about 1 of 4th of "w". Needless to say, variable-width font is more close the nature of hand-writing. Emacs has a built-in variable-pitch-mode that could change the font.

But will it make any different to my writing? I am not sure at this moment, but I would like to have a special font that I solely use in writing. The link between the font and my write mind will gradually become firm, and eventually increase my productivity in writing.

Sentence Highlight

Writing requires thinking and concentration. People have their own tips that help them to stay focus and get writing done, it may relates to a place, time or tools.

I tired many tips, like mediate before write, drink coffee, cut off internet but none of them works very well, the effects seems random. One problem I have in writing is that I jump between the sections quite often.

I tried to highlight the one sentence at a time so that I can focus on the one I am writing. I found hl-sentence package does exactly what I want. Also, I followed the author's suggestion and tweak the configuration to blur the other sentences to reduce the noise.

The current setting has two folder and helps me in a way that I can focus naturally: I don't need to force myself not looking other sentence.

The sentence highlight feature also has an big impact on my writing process by making the editing easier. One thing I want to achieve is to have proper length for each sentence/paragraph: If it is too short, I will merge it. If it is too long, I will break up into short sentences. The highlights give me a sense of the length visually which I used to get by reading or counting. To check how many sentences exactly for each paragraph, I move the cursor to end of a sentence by M-e, and then count how many flashes I have to reach the end of a paragraph.

Wrap-up

I am fairly happy about the nanowrimo, hl-sentence and variable-pitch mode and the powerful Emacs. Thanks to all the authors who wrote the scripts, because of their quality work, many things work out of box and I am able to have an seamless integration to the current workflow. It has became more efficient and productive, and makes me believe the Emacs is the best writing tool for me.

Which program do you use for writing? which feature do you like most?

How Do I Build This Blog

2014-12-17T00:00:00+00:00

Why I Learn Jekyll/Blog
Process Raw Data
Analysis 1 - Mixture of levels
Analysis 2 - Time distribution
Analysis 3 - Benefits
Conclusion

To Reader

The learning skill becomes more and more important nowdays because there are just so much to learn, either from daily job or personal interest. But have you ever thought about the way you learn or how good is your learning skill? In this article, I want to share my experience in learning how to build this blog using Jekyll and how I exam the way I learn via data and statistical analysis. It is worth reading if you:

want to improve your learning skill,
are trying to learn Jekyll or want to build a personal website,
are interested in quantified-self project.

Why I Learn Jekyll/Blog

All my high-school classmate known that I am really bad at writing. Things becomes worse when I was in university studying mathematics. I start to void writing at all the cost: I deliberately volunteed the do the coding or maths bit and let others do the writing. Everybody in the group seems like me.

During my study in Warwick University, I fancy the Statistics and really enjoy telling people the relationship I between all sort of facts. Then I come across The Guardian's Data Journalism program which is a term in use since 2009/2010, to describe a journalistic process based on analyzing and filtering large data sets for the purpose of creating a news story¹. I found it is really cool! I tried to analyst a new dataset from World Bank and want to write an article but when I sit down my mind just completely blank. I managed to write a few paragraph but it was really awaful. That was the first time that I thought I wish I have a proper writing skill. I won't bother to learn because I planed to go back to China soon but I decided to work in the UK at the last minute.

Last week, I was doing a statistical consutlantcy project. I was do the analysis, writing code in the morning then try to write it up in the afternoon. I realise it pretty easy for me to do the analysis and coding, but it was really a pain to write it up, but I do enjoy it. So I decided to build a personal blog so that I can practise my writing skill and at the same time, to promote the usage of statistics in daily life.

I have no prior knowledge of building a website and spent couple of evenings/weekends sitting in front of computer at library/caffee and try to learn. This project accros two months and takes me about 20 hours in total to finally build this blog. Motivation really is crucial to learning. What make this learning expeirence really unique is that I have data about my learning.

Process Raw Data

I have a very good habit: I record every single tasks I do in terms of how much time I spent and what I did. Take this task for example,

PM (strcture)
SCHEDULED: <2014-12-11 Thu 17:45>
:LOGBOOK:  
CLOCK: [2014-12-11 Thu 17:41]--[2014-12-11 Thu 18:28] =>  0:47
:END:      
:PROPERTIES:
:Effort:   0:45
:END:
[2014-12-10 Wed 22:28]
wait for 4 minutes 
- [ ] go though all the headlines,
- [ ] group into few categories, like 1) learn, 2) apply, 3) improve etc.
- [ ] start to edit

This note includes all the info I need to know:

I want to do it On Wed and estimate it takes 45 minutes to compltete
I planed to do on Thurs,
I started at Thur 17:41, 4 minutes before the scheduled time,
It takes me 47 minutes to do the job, 2 minutes longer then expected.
what i did, the main body

I gathered all the notes that are relevant to this project and accumulate the time I spent for each sub-tasks. It can be summaries as a table:

Table 1: Clock summary at [2014-12-10 Wed 21:42]
Headline	Time
Total time	17:22
TODO blog	17:22
DONE Jekyll official guide on github…		1:22
workflow		1:45
TODO how to change the look of html		2:11
org-jeykll workflow (final)		1:26
DONE blog, does not looks good on…		0:10
discovery jeykll template		1:57
DONE Jeykll disaster		1:00
NEXT tweak jkyll (add social content…		1:17
DONE Jekyll (disqus and google…		0:25
Jekll,		2:23
Jekyll general search		0:53
NEXT intro to jekyll		0:28
Jekyll code highlight		1:03
DONE Jekyll - non-doing action		1:02

It tells that I spent 17 hours and 22 minutes in total on this project. The first question is: what does this 17 hours mean to me? I spent 500hours plus in playing video games between 2013-2014 and I really don't get nothing out of it. I really enjoy this learning experience because I have a definitive goal I want to achieve, then I put effort and I can get some results.

The rest of the table tells me tasks I did in chronological order. The first task was to read the official Jekyll guide and I spent 1 hour 22 minutes. This table looks really awfully and gives you an insight of what a real world data looks like. I don't really know what to do with it. So I have to refers to the way I learn as a child.

From our education system, the way we are learning is that we start on a basic level, we study the topic, apply it, make mistakes, correct it, and continue this process to the next level. It gives me an idea to define "levels". So I skim the whole project, from beginning to end, and grouped the data into five categories:

Basis: the foundation of Jekyll, website and HTML language,
Features: he extended features provided by Jekyll, for example, add social network link, add a discussion/comments,
Workflow: integrate the publishing process to my current workflow and try to automate as much process as possible,
Try: try to impelement something new that for my own needs, or try other people ideas,
Fix: fix problems along the way I build up the website

Analysis 1 - Mixture of levels

There is a linear increasing level, and one dependence the previos level. Ideally, the most cost-effective approach for a person to learn, is to focus on step, master it, and then goes to the next. This is the way of our education system. But it won't be the case of a self-study project, which is most likely to be interest-driven. it wold be very interesting to see that how I was jumping between these five levels. I sort out the timeline for each tasks I did and plot the time with levels.

Figure 1: timeline view

The x-axis is the time line in minutes for this project and y-axis is the level I defined. I can tell which level I am in a studying time. For example, for the first 200 minutes I was studying the basis knowledge, then I spent 35 minutes on Feature and so on.

I am very surprised that I went to workflow level so early, in about 20% of this project. It is actually make a lot sense because this project across 2 months and having a workflow that suitable for me really accelerate this project because It takes no time to pick up after leaving this project for few days.

There was a time that the website is broken and I cannot figure out why. It turned out I missed spelled an configuration file.

Finally I spent some time on googling about how other people use Jekyll and tried quit a few and never goes back to lower level.

Analysis 2 - Time distribution

It would be also interesting to see how long I spent on each levels, i.e.

Figure 2: time distritbuoin

Level	Time (Min)	Percentage
Basis	184	0.18
Feature	276	0.26
Workflow	191	0.18
Fix	70	0.07
Try	321	0.31

The pie chart should be read in clocking-wise direction and it is ordered by levels. I spent 3 hours in Basis level and to be honest I understand much about Jekyll. It is interesting to see that I spent 8% more time on the expanded features than the basic knowledge, i.e. using it rather study the knowledge.

Thanks to all the volunteers that working on connect Jekyll and Org mode. I only spent 3 hours in setting up the workflow that includes write article in Org mode (plain text), convert it to HTML web page, and then upload to my blog. Jekyll is a sophisticated and well tested software that it can be configured easier and I didn't running into any problems.

The rest 5.4 hours was the most inefficient in this project. At that time I was keen to include Table of Content and Code Syntax Highlight in my blog. I did a lot further research for solutions and tried a lot. But none fits well with the foundation I have built up. Some is buggy and create more problems. If you really need these feature, see this blog.

Analysis 3 - Benefits

The simple question: is it worth spending 18 hours building a website side? In timewise, I have spent 20 hours on writing, but this number does not count, because I can write without a blog at all. But I feel that having this blog promote me writing because:

the writings has their destination. It is not a digital file in my computer that I will forget in few weeks or a page in a notebook I hardly look back with potentially lose. All my writings will be in this single website that has a unique address that everybody can visit it whenever or whatever they are.
the aim for writing has been changed. It is not only to express myself, like a daily, but to share an journal to other. I have to consider the reader's feel, will they like it? will they understand it? So, the writing becomes more proactive, more thinking on human relationship, and thus more fun.
Writing is linked to this website, which is linked to quantifies-self project and Emacs/org-mode, which then linked back to statistics and programming, which are the two main passion of me. Writing is not something that occurs to my mind one day and then I swear I will master it, but something in my passion network and extended my passion to another area.

Conclusion

[2014-12-19 Fri 13:21] First, quantify my time on learning is

Words: 1657, Write: 5 Hours

Learning How to Learn: Powerful mental tools to help you master tough subjects The Data Journalism Handbook

Footnotes:

wiki

Yi Tang

Giving Qwen 3.6 35B Vision

Footnotes

Multiple Working Emacs

Installation

Daemon

GUI

Not Perfect But Close

Rebate Architrave

Terminating Ethernet Cable At Height For CCTV Cameras

On the Ground

At Height

Footnotes

Extend Ethernet Cable

Jelly Crimps

Inline Coupler from Kenable

Re-discovery the Ancient Info Documentation System in the Age of LLM

Table of Contents

Travel with Info

Why Info is not Popular?

dir the Index File

Setup Info in MacOS

Filter Ledger Transactions using Tags

Setup ssh-agent Systemd Service for Emacs

Problem Statement

Implementation

Alternatives

Footnotes

Retiring Raspberry Pi 4 as Home Server and NAS

Table of Contents

Good Start for Self-Hosting

Lack of NAS Capacity

Looking for a Successor

Unexpected

Setting up z170a

Power Consumption

Footnotes

Use Ledger-Cli to Track DIY Project Expenses

Table of Contents

Personal Technical Challenge

Baby Steps

Why? - Effort Estimation

Footnotes

Finding Highly Correlated Features

Table of Contents

Motivation

Implementation

Parameterisation

Less Excel, More R/Python in Emacs

Table of Contents

Excel Is Great

But

Emacs Has More To Offer

Blog in Emacs - Use Jekyll's Draft Mode

Why?

Implementation

Blog in Emacs - Work with Images

Learn in Emacs - Building Up Vocabulary

Table of Contents

WHY?

Workflow for Building Vocabulary

Revision on Mobile Devices

Org-mode Based Simple Study Strategies

Emacs Lisp Implementation

Footnotes

Atomic Habit in Emacs - Keep Git Repos Clean

Table of Contents

Why?

Emacs Lisp Helper

Practise

Footnotes

GPG in Emacs - Functions to Decrypt and Delete All

Table of Contents

Motivation

Emacs Lisp Implementation

Bash Implementation

Footnotes

GPG in Emacs - First Step Towards Data Security

Table of Contents

WHY?

`dir` the Index File

Emacs’ Way: `dired-rsync`

Setup for `dired=rsync`

Enhance `dired-rsync` with compilation mode