The good thing about Emacs is that you can always tweak it to suit
your needs. For years I’ve been doing it for productivity reasons. Now
for the first time, I’m doing it for health reasons.
Life can be sht sometimes, when I was in my mid 20s, I was reshaping
every aspects of my life for good. But optician told me my vision can
only get worse. I wasn’t paying much attention, busy with my first
job and learning.
Last month, I was told my right eye’s vision got whole point worse,
whatever that means. Now I’m wearing a new pair of glasses, seeing the
world in 4K using both eyes, noticing so much details. It makes the
world so vibrate and exciting. It comes with a price though, my eyes
get tired quickly, and it become so easy to get annoyed by little
One of them is switching windows in Emacs. Even though I am in the
period of calibrating to the new glasses, I decided to take some
Depends on the complexity of the tasks, I usually have about 4-8
windows laid on my 32 inch monitor. If that’s not enough, I would have
an additional frame of similar windows layout, doubling the number of
windows to 8-16.
So I found myself switching between windows all the time. The action
itself is straightforward with ace-window.
The process can be breakdwon into five steps:
Invoke ace-window command by pressing F2 key,
The Emacs buffers fade-in,
A red number pops-up at the top left corner of each window,
I press the number key to switch the window it associates with,
After that, the content in each Emacs buffer are brought back.
This approach depends on visual feedback - I have to look at the
corner of the window to see the number. Also, the screen flashes
twice during the process.
I tried removing the background dimming, increase the font size of the
number to make it easier to see, and bunch of other tweaks.
In the end, my eyes were not satisfied.
So I started looking for alternative approaches and found windmove
which is built-in.
The idea is simple - keep move to the adjacent window by move left,
right, up, or down until it arrives at the window I want.
So it uses the relative location between windows instead of assigning
each window a unique number and then using the number for switching.
Is it really better? Well with this approach, I use my eyes a lot less
as I do not have to look for the number. Plus, I feel this is more
nature as I do not need to work out the directions, somehow I just
know I need to move right twice or whatever to get to the destination.
The only issue I had so far is the conflicts with org-mode’s
calendar. I like the keybinding in org-mode, so I disabled windmove
in org-mode’s calendar with the help from this stackoverflow question.
The following five lines of code is all I need to use windmove.
I created an git branch for switching from ace-window to
windmove. I would try it for a month before merge it into master
Back to where it started
After using it for few days, I realised this is the very package I
used for switch windows back in 2014 when I started learning Emacs. I
later then switched to ace-window because it looks pretty cool.
Life is changing, my perspectives are changing, so is my Emacs
configuration. This time, it goes back to where I started 8 years ago.
After 3 months using my brand new MacBook Pro 14 M1 Pro, one of the
USB-C port stopped working. I will have to send it back, not sure what
Apple will do with it but I can't bear the risk of losing data. So I
need a backup.
In fact, I need to backup regularly for situation like this so that's
why I worked on it.
Wireless Backup Solution
The easiest solution is to get a USB-C portable SSD, plug it into my
laptop and open Time Machine to start back up, do it once a week and
call it a day.
But I'm reluctant to add more devices to my already cluttered home
lab. There are a few hard drives in the drawers, it would be good to
So I decided to set up a Time Machine backup solution using on my
Raspberry Pi 4. The benefits are
no additional costs, save me about £50-£100
no need to buy new stuff, so fewer things to care of
wireless backup to keep my desk clean
Later I realised the benefits of having a wireless backup is
overlooked. It can backup anytime and anywhere in my house. Also,
because of convenience, I can have more granular backups - instead of
weekly backup, I have hourly backup without getting the cables and
hard drives. I do less but get more value out of it.
The only concern I had was the speed. It turns out with SAMBA 3
protocol, I can get 55 MB/s write speed and 40 MB/s read speed from
laptop to Raspberry Pi. So in theory, it would take around 2.5 hours
to backup my 500 GB laptop. It might be a lot but only for the first
backup, the subsequent incremental backup would be much simpler and
faster, for example, as of now, the Time Machine completed a new
backup within 3 minutes in the background without my notice.
A portal USB-C SSD can finish the backup within minutes but it's an
overkill for an ordinary user like me and it's inconvenient.
So I'm satisfied with the current solution.
Set Up Raspberry Pi
I read a few guides on setting up Raspberry Pi for Time Machine, and I
found this guide most accurate and useful.
One thing I noticed is the AFP (Apple File Protocol) is deprecated, so
make sure you use SAMBA as the protocol.
Additionally, I followed this stack overflow answer to auto-mount the
SAMBA server so that every time I reboot my laptop, the Time Machine
will be ready to back up.
Time Machine Backup frequency
By default, Time Machine does hourly backup.
If you feel hourly backup is not necessary, you can change it by
updating this file
for example, to change the frequency from hourly to daily backup,
change the interval value from 3600 to 43200.
In the end, I left it with the default hourly backup so it does many
small backup hourly instead of one big backup daily.
Backup for Backups
After couple of hours of work, I managed to get a wireless backup
solution for my laptop so I won't have to worry about data loss. Plus
I can time-travel files at hourly intervals.
One concern that occurred to me was the backup sits on my local hard
drive. If the hard drive died, I would lose all my backups.
To solve that problem, I will have to go through the rabbit hole of
doing backup for backups, or backup to a remote location or cloud, or
setup a Raspberry Pi RAID.
At the moment, I'm not very concerned - I have Apple iCloud to back up
my photos, videos, notes etc and I use GitHub to host my org-files and
code. So having a backup for backups is not necessary for me for now.
I live in Emacs entirely apart from using browser for googling. Having
an Emacs server running on the background makes Emacs available all
the time. So I won't worry about closing it accidental.
It is not hard to do that, just run
in command line to start the Emacs server. It will load user
configuration file as usual. Then run
to open an Emacs GUI instance that uses the Emacs server. That's
how I have been doing for a while.
An better approach is using systemd. It is the services manager of
Linux. Whenever my Debian 11 laptop boot up, systemd would start a
bunch of services in parallel, for example, Networking manager
connects WIFI, Bluetooth connects wireless keyboard so everything
would be ready after I login. And I want Emacs to be ready as well.
I can achieve that by simply having an shell script automatically
running after login. But there are benefits of using systemd. It
has bunch of sub-commands for managing services, for example,
checking logs, status etc.
It's a nice tool to have, I can use it for example Jupyter Notebook
That's why I pulled the trigger and spent 2 hours in implementing
and testing it. Here's the technical bit.
How to Implement As Systemd Service
In order to use systemd to manage Emacs server, I firstly need a
configuration file (which is called unit file). Debian Wiki
provides a short description of the syntax and parameter of unit
I found an simple one in Emacs Wiki. It looks like this
service, in this case it runs /usr/bin/emacs --daemon command.
it tells systemd what to do when shutting down Emacs
service, in this case it runs /usr/bin/emacsclient --eval
If you are using an Emacs built in a difference directory, you have
to change /usr/bin/emacs to wherever your Emacs is located.
Then save the configuration file as
After that run
so systemd would copy the configuration file into central places
and it would start Emacs service at boot time.
To run Emacs service right now, use
This is what I see in my console
emacs.service - Emacs text editor
Loaded: loaded (/home/yitang/.config/systemd/user/emacs.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-06-14 09:12:26 BST; 24h ago
Main PID: 5222 (emacs)
Tasks: 5 (limit: 19027)
CPU: 3min 25.049s
├─ 5222 /usr/bin/emacs --daemon
└─16086 /usr/bin/aspell -a -m -d en_GB -p /home/yitang/git/.emacs.d/local/ispell-dict --encoding=utf-8
Jun 14 09:11:57 7270 emacs: No event to add
Jun 14 09:11:57 7270 emacs: Package dash-functional is obsolete; use dash 2.18.0 instead
Jun 14 09:12:01 7270 emacs: Loading /home/yitang/git/.emacs.d/config/org-mode.el (source)...done
Jun 14 09:12:01 7270 emacs: Loading /home/yitang/git/.emacs.d/config/refile.el (source)...
Jun 14 09:12:01 7270 emacs: Loading /home/yitang/git/.emacs.d/config/refile.el (source)...done
Jun 14 09:12:01 7270 emacs: Loading /home/yitang/git/.emacs.d/config/scripting.el (source)...
Jun 14 09:12:26 7270 emacs: Loading /home/yitang/git/.emacs.d/config/scripting.el (source)...done
Jun 14 09:12:26 7270 emacs: Loading /home/yitang/git/.emacs.d/load_config.el (source)...done
Jun 14 09:12:26 7270 emacs: Starting Emacs daemon.
Jun 14 09:12:26 7270 systemd: Started Emacs text editor.
Enhance User Experience
For far, I have the following two tweaks to make the usage of systemd
The Emacs server is started using my own account, so it doesn't have
the sudo privilege. In order to edit files that requires sudo
permission, simple open the file in Emacs, or in command line with
then type M-x sudo inside Emacs, type the sudo password. If the
password is correct, I can edit and save the file as sudo user.
The customised shell configuration in .bashrc are loaded when
opening an interactive shell session. So the Emacs server managed by
systemd would not have the environment variables, alias, functions or
whatever defined in .bashrc.
This problem can solved a lot easier on the Emacs side, by using
exec-path-from-shell package. It will ensure the environment
variables inside Emacs are the same as in the user's interactive
Simply put the following in your .emacs would do the trick.
Start Emacs Server Before Login?
The systemd services under my account would only start after I
login. Because I have tons of Emacs configuration, I still have to
wait few seconds before Emacs server is ready. So it would be awesome
to have the Emacs server starting to load before I login.
This doesn't seems to be simple to implement, because technically,
it would require the Emacs server to be defined on system level,
but it will load files in my personal home drive without me being
logged in. It might be still okay since I'm the sole user of my
laptop, but I have to tweak the permissions and would probably end
up with non-secure permission setting.
Avito Demand Prediction Challenge asks Kagglers to predict the
"demand" likelihood of an advertisement. If an listed 2nd-hand Iphone
6 is selling for £20,000, then the "demand" is likely to be very low.
This is the my first competition to build model using tabular data,
text, and also images.
I teamed up with Rashmi, Abhimanyu, Yiang, Samrat and we finished at
22 among 1917 teams. So far, I have four silver medals and my rank is
542 among 83,588 Kaggler.
This is an interesting competition for me. I was about to quit this
competition and Kaggle because of other commitments in life/work. Just
one day before team merge deadline, Rashmi asked me to join, at that
time, my position is 880-th, about 50%, and Rashmi's team is about
82-th. So I decided to join and finish this competition which I
already spent about many hours.
Final Ensemble Models
As part of this team, I worked on final ensemble models. Immediately
after join, i completed 5 tasks:
make sure everyone uses the same agreed cross validation schema.
This is essential for building ensemble model.
provide model_zoo.md document to keep track of all level 1
models, their train/valid/lb scores, feature used, and file path
to their oof/test prediction.
write merge_oof.py to combine all oof/test predictions together.
write R scripts for glmnet ensemble.
write python scripts for LightGBM ensemble.
Once new model is built, other team member update the model_zoo.md
and upload the data to a private github repo. Then I update the
merge_oof.py to include new models' result, and run glmnet and
LightGBM ensemble. We had this ensemble workflow automated so it
takes little effort to see the ensemble model's performance.
I spent some times analysing the coefficients/weights of L1 model
and tried to exclude models with negative and lower weights. To my
surprise it doesn't help at all. The final submission is a glmnet
ensemble with 41 models (lgb + xgb + NN).
Also, LightGBM ensemble has much better cv score but the LB score
is worse. I suspect it is because there are leakage in L1 models
and glmnet is more robust to leakage since it's linear model.
Unfortunately, there's no enough time to identify which models have
This is my 2nd time work in a team, although there's a lot space
for improvement collaborating when compared with a professional
data scientist team but as night/weekend project, we have done a
really good job as a team.
The setup for collaboration:
Slack for discussion. we have channel for general,
final_ensemble, random for cat photos etc.
we also used Slack for sharing features which i personal don't
Private github repo for sharing code and oof/test predictions.
Monday.com for managing tasks. it gives a nice overview of what
everyone's up to.
we tried very hard to get a gold, but other teams work even more
harder. At one point we were at 17, and finished at 22.
Some Kagglers to Avoid
Finally, when we waited 1 hour for the final deadline, we had a
lovely discussion about our past disqualification experience. We
were all shocked when we were at different team in Toxic
competition but team up with the same person. We shared their
person's multiple Kaggle accounts and added to our personal
Data processing time will becomes longer and longer as the increasing
rate of data volumes. Users may check-in frequently to see the whether
it is finished. Most of the time they will found it hasn't, doing this
the users make contact switch which break the flow of whatever the
using was doing.
Sometimes the user can't stop doing so, either because they are
impatient, or because they really have a deadline to catch. Also, with
less likelihood, they might find errors in the processing, either
because of the QC check fails, or running out of computational
Giving this fact, it really makes sens to have your program actively
inform the user on the process so that they doesn't need to check-in
at all. Because users will be notified immediately whenever the
whole progressing is completed, or there's error that the user needs
to take action onup.
This blog posts walk though the basics of sending Emails in Python,
composing and sending out Emails. Each component is broken down into
small piceses. It helps you debug/tests Email program, and personalise
your Emails. In the end, you should be able to build a email robot.
Before going into the technical details, you have to check that you
are able to send out emails. You need the
[ ] SMTP server,
[ ] user name,
[ ] password,
[ ] port number, and
[ ] communication protocol.
You could easily found out these information from your Email service
provider. An example of Gmail is at Here.
To check if you have all the information correct, run the following
snippet. It will try to send out an empty Email to yourself. Make
sure you fill in the username and password before hit go.
If you don't see error messages, that's great, you are all set.
There should be an Email in your Inbox. It has no subject and for
testing in the main body.
If you do, you need to double check the information, and try
again. If you are sure that the information are correct, btu still
can't send out, check your network configuration, maybe the firewall
block the connection.
Once you are able to send out an empty email the next is to compose
an full Email.
Compose An Email in Python
An Email is consisted of multiple parts, the Subject, Body,
attachments, Signature, and also some meta-data including from, to,
Firstly, create an object of MIMEMutiple() class. It will be the
building block of your Email. The approached described here is to
add each components into it.
Email meta data
Start with meta data.
For plain text, simply add
You could also try to compose an complex HTML email in Python, or
use diffenret tool to generate the HTML and import in Python.
You can attach files of any type in an Email, specially, for image,
you could use MIMEImage, and for audio, you could use
But you don't have to be specific. MIMEApplication would be
sufficeincy for all cases. It will configure the file type of the
attached file. Use it as follows:
Put Everything Together
Wrap everything in a Class
At JBARML, we are planing to send users emails for a progress
update. Many of the data generating process are lured together and
automated by a workflow manager. The whole programcan takes upto
weeks to complete. Actively sending update progeress to user is
much more senssible then user loggin in to a remote machien and
check it now and then.
In this case,
email to notify user the progress of the data