I have this setup in Linux using systemd, as in my previous blog post.

In MacOS, the launchctl is the service manager. It provides a user interface to list, start and stop services.

To build an Emacs server, create a plist file in ~/Library/LaunchAgents folder. In my case, I named it emacs_work.plist.

 1: # cat ~/library/LaunchAgents/emacs_work.plist
 2: <plist version="1.0">
 3:   <dict>
 4:     <key>Label</key>
 5:     <string>emacs_work</string>
 6:     <key>ProgramArguments</key>
 7:     <array>
 8:       <string>/opt/homebrew/opt/emacs-plus@29/bin/emacs</string>
 9:       <string>--fg-daemon=work</string>
10:     </array>
11:     <key>RunAtLoad</key>
12:     <true/>
13:     <key>KeepAlive</key>
14:     <true/>    
15:     <key>StandardOutPath</key>
16:     <string>/tmp/emacs_work.stdout.log</string>
17:     <key>StandardErrorPath</key>
18:     <string>/tmp/emacs_work.stderr.log</string>
19:   </dict>
20: </plist>

It is an extension of Emacs Plus' plist file. I made a few changes for running two Emacs servers: one for work (data sciences, research) and one for personal usage (GTD, books). Taking the "work" server as an example, the important attributes of the plist configuration file are:

Line 5

The unique service name to launchctl

Line 8

The full path to the Emacs program. In my case, it is /opt/homebrew/opt/emacs-plus@29/bin/emacs

Line 9

The "--fg-daemon" option set the Emacs server name to "work". Later I can connect to this server by specifying "-s=work" option to emacsclient

Line 13

The KeepAlive is set to true so it keeps trying to restart the server in case of failures

Line 16 and 18

The location of standard output and error files. They are used to debug. Occasionally I have to check those files to see why Emacs servers stopped working, usually because of me introducing bugs in my .emacs.d.

With the updated plist files in place, I start the Emacs servers with

launchctl load -w ~/Library/LaunchAgents/emacs_work.plist
launchctl load -w ~/Library/LaunchAgents/emacs_org.plist

The launchctl list | grep -i emacs is a handy snippet that lists the status of the services whose name includes "emacs". The output I have right now is

It shows both Emacs servers are running fine with exit code 0.

I can now open a Emacs GUI and connect it to the "work" Emacs server by running emacsclient -c -s work &. The -c option

In MacOS, I found it is natural to open applications using Spotlight, for example, type ⌘ + space to invoke Spotlight, put "work" in the search bar, it narrows the search down to "emacs_work" application, and hit return to finalise the search. It achieves the same thing as the command above but can be used anywhere.

I uploaded a demo video on YouTube to show it in action. You might want to watch it at 0.5x speed because I typed so fast...

To implement this shortcut, open "Automator" application, start a new "Application", select "Run Shell Script", and paste the following bash code

/opt/homebrew/opt/emacs-plus@29/bin/emacsclient \
    --no-wait \
    --quiet \
    --suppress-output \
    --create-frame -s work \
    "$@"

and save it as emacsclient_work in the ~/Application folder.

Essentially, the bash script above is wrapped up as a MacOS application, named emacsclient_work and the Spotlight searches the application folder by default.

To train the model, the price and volume data are transformed into black-white images which is just a 2D matrix with 0s and 1s. For just around 100 stocks' pricing history, there are around 1.2 million images in total.

I used the on-the-fly imaging process during training: in each batch, it loads pricing data for a given stock, sample one day in the history, slice a chunk of pricing data, and then convert it to an image. It takes about 0.2 milliseconds (ms) to do all that, so in total it takes 4 minutes to loop through all the 1.2 million images.

%%timeit 
df = MarketData(DATA_DIR)['GOOGL']
imager = ImagingOHLCV(img_resolution, price_prop=price_prop)
img = imager(df.tail(5))

1.92 ms ± 26.9 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

To train 10 epochs, that's 40 minutes in loading data. To train one epoch on the full dataset with 5,000 stocks, that's 200 minutes in loading data alone!

PyToch utilises multiple processing in loading the data using CPU while training using GPU. So the problem is less severe, but I'm using the needle, the deep learning framework we developed during the course, it does have this functionality yet.

During training using needle, the GPU utilisation is only around 50%. After all the components in the end-to-end are almost completed, it is time to train with more data, go deeper (larger/more complicated morel), try hyper-parameters tuning etc.

But before moving to the next stage, I need to improve the IO.

In the image above, there are a lot of black pixels or zeros in the data matrix. In general only 5%-10% of pixels are white in this dataset.

So my first attempt was to use scipy's spare matrix instead of numpy's dense matrix: I save the sparse matrix, loaded it, and then convert it back to a dense matrix for training CNN model.

%%timeit
img_sparse = sparse.csr_matrix(img)
sparse.save_npz('/tmp/sparse_matrix.npz', img_sparse)
img_sparse_2 = sparse.load_npz('/tmp/sparse_matrix.npz')
assert np.all(img_sparse_2 == img)

967 µs ± 4.99 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

It reduces the IO time to 1ms, so about half of the time, not bad, but I was expecting a lot more as the sparseness is high.

Then I realised the data behind images is just 0 and 1, in fact, a lot of zeros, and only some are 1. So I can ignore the 0s and only need to save those 1s, then reconstruct the images using those 1.

It is so simple that numpy has functions for this type of data processing already. The numpy.packbites function converts the image matrix of 0 and 1 into a 1D array whose values indicate where the 1s are. Then the numpy.unpackbits does the inverse: it reconstructs the image matrix by using the 1D location array.

This process reduces the time of loading one image to 0.2 milliseconds, that's 10 times faster than the on-the-fly method with only a few lines of code.

%%timeit 
temp_file = "/tmp/img_np_bites.npy"
img_np_bites = np.packbits(img.astype(np.uint8))
np.save(temp_file, img_np_bites)
img_np_bites = np.load(temp_file)
img_np_bites = np.unpackbits(img_np_bites).reshape(img.shape)
assert np.all(img_np_bites == img)

194 µs ± 3.95 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

Another benefit is the file size is much smaller: it is 188 bytes compared to 1104 bytes using sparse matrix. So it takes only 226MB of disk space to save 1.2 million images!

Path('/tmp/img_np_bites.npy').stat().st_size, Path('/tmp/sparse_matrix.npz').stat().st_size

188, 1104

It takes a couple of minutes to generate 1.2 million files on my Debian machine. It is so quick! But then I release this approach is not scalable without modification because there's a limited number of files the OS can accommodate. The technical term is Inode. According to this StackExchange question, once the filesystem is created, one cannot increase the limit (Yes, I was there).

Without going down to the database route, one quick workaround is to bundle the images together, for example, 256 images in one file. So later in training, load 256 images in one go, then split them into chunks. Just ensure the number of images is a multiple of the batch size used in training so I don't have to deal with unequal batch sizes. Since those bundled images are trained together, it reduces the randomness of SGD, so I won't bundle too many images together, 256 sounds about right.

The LSP and other tools can cause problems when they are monitoring folders with a large number of files. Moving them out of the project folder is the way to go so Emacs won't complain or freeze.

So I implemented what I call PoorMan CI. It is a semi-automated process that gives me some benefits of the full CI. I tried hard to refrain from doing anything fancy because I don't have time. The final homework is due in a few days. The outcome is simple yet powerful.

The PoorMan CI consists of two parts:

1. a bunch of bash functions that I can call to run the tests, capture the outputs, save them in a file, and version control it

   For example, wrap the below snippet in a single function

pytest -l -v -k "not training and cuda" \
       > test_results/2022_12_11_12_48_44__fce5edb__fast_and_cuda.log
git add test_results/2022_12_11_12_48_44__fce5edb__fast_and_cuda.log

1. a log file where I keep track of the code changes, and if the new change fixes anything, or breaks anything.

   In the example below, I have a bullet point for each change committed to git with a short summary, and a link to the test results. The fce5edb and f43d7ab are the git commit hash values.

   - fix grid setup, from (M, N) to (P, M)!
   [[file:test_results/2022_12_11_12_48_44__fce5edb__fast_and_cuda.log]]
   
   - ensure all data/parameters are in the right device. cpu and cuda, all pass! milestone.
   [[file:test_results/2022_12_11_13_51_22__f43d7ab__fast_and_cuda.log]]
   

As you can see, it is very simple!

It changed my development cycle a bit: each time before I can claim something is done or fixed, I run this process which takes about 2 mins for two fast runs. I would use this time to reflect on what I've done so far, write down a short summary about what's got fixed and what's broken, check in the test results to git, update the test log file etc.

It sounds tedious, but I found myself enjoying doing it, it gives me confidence and reassurance about the progress I'm making. The time in reflecting also gives my brain a break and provides clarity on where to go next.

During my few hours of using it, it amazes me how easy it is to introduce new issues while fixing existing ones.

I don't have to use Org-mode for this, but I don't want to leave Emacs :) Plus, Org-mode shines in literate programming where code and documentation are put together.

This is actually how I implemented it in the first place. This section is dedicated to showing how to do it in Org-mode. I'm sure I will come back to this shortly, so it serves as documentation for myself.

Here is what I did: I have a file called poorman_ci.org, a full example can be found at this gist. An extract is demonstrated below.

I group all the tests logistically together into "fast and cpu", "fast and cuda", "slow and cuda", "slow and cuda". I have a top level header named group tests, Each group has their 2nd-level header.

The top header has a property drawer where I specify the shell session within which the tests are run so that

* grouped tests
:PROPERTIES:
:CREATED:  [2022-12-10 Sat 11:32]
:header-args:sh:    :session *hw4_test_runner* :async :results output :eval no
:END:

1. it is persistent. I can switch to the shell buffer named hw4_test_runner and do something if needed
2. it runs asynchronically on the background

All the shell code block under the grouped tests inherits those attributes.

The first code block defines variables that used to create a run id. It uses the timestamp and the git commit hash value. The run id is used for all the code blocks.

#+begin_src sh :eval no
wd="./test_results/"
ts=$(date +"%Y_%m_%d_%H_%M_%S")
git_hash=$(git rev-parse --verify --short HEAD)
echo "run id: " ${ts}__${git_hash}$
#+end_src

To run the code block, move the cursor inside the code block, and hit C-c C-c (control c control c).

Then I define the first code block to run all the tests on CPU except language model training. I name this batch of tests "fast and cpu".

#+begin_src sh :var fname="fast_and_cpu.log"
fname_full=${wd}/${ts}__${git_hash}__${fname}
pytest -l -v -k "not language_training and cpu" \
     2>&1 | tee ${fname_full}
#+end_src

1. It creates the full path of the test results. The fname variable is set at the code clock header, this is a nice feature of Org-mode.
2. pytest provides an intuitive interface for filtering tests, here I use "not language_training and cpu".
3. The tee program is used to show the outputs and errors and at the same time save them to a file.

Similarly, I define code blocks for "fast and cuda", "slow and cpu", "slow and cuda".

So at the end of the development cycle, I open the poorman_ci.org file, run the code blocks sequentially, and manually update the change log. That's all.