After 3 months using my brand new MacBook Pro 14 M1 Pro, one of the USB-C port stopped working. I will have to send it back, not sure what Apple will do with it but I can't bear the risk of losing data. So I need a backup.

In fact, I need to backup regularly for situation like this so that's why I worked on it.

The easiest solution is to get a USB-C portable SSD, plug it into my laptop and open Time Machine to start back up, do it once a week and call it a day.

But I'm reluctant to add more devices to my already cluttered home lab. There are a few hard drives in the drawers, it would be good to utilise them.

So I decided to set up a Time Machine backup solution using on my Raspberry Pi 4. The benefits are

1. no additional costs, save me about £50-£100
2. no need to buy new stuff, so fewer things to care of
3. wireless backup to keep my desk clean

Later I realised the benefits of having a wireless backup is overlooked. It can backup anytime and anywhere in my house. Also, because of convenience, I can have more granular backups - instead of weekly backup, I have hourly backup without getting the cables and hard drives. I do less but get more value out of it.

The only concern I had was the speed. It turns out with SAMBA 3 protocol, I can get 55 MB/s write speed and 40 MB/s read speed from laptop to Raspberry Pi. So in theory, it would take around 2.5 hours to backup my 500 GB laptop. It might be a lot but only for the first backup, the subsequent incremental backup would be much simpler and faster, for example, as of now, the Time Machine completed a new backup within 3 minutes in the background without my notice.

A portal USB-C SSD can finish the backup within minutes but it's an overkill for an ordinary user like me and it's inconvenient.

So I'm satisfied with the current solution.

I read a few guides on setting up Raspberry Pi for Time Machine, and I found this guide most accurate and useful.

One thing I noticed is the AFP (Apple File Protocol) is deprecated, so make sure you use SAMBA as the protocol.

Additionally, I followed this stack overflow answer to auto-mount the SAMBA server so that every time I reboot my laptop, the Time Machine will be ready to back up.

By default, Time Machine does hourly backup.

If you feel hourly backup is not necessary, you can change it by updating this file

/System/Library/LaunchDaemons/com.apple.backupd-helper.plist

for example, to change the frequency from hourly to daily backup, change the interval value from 3600 to 43200.

In the end, I left it with the default hourly backup so it does many small backup hourly instead of one big backup daily.

After couple of hours of work, I managed to get a wireless backup solution for my laptop so I won't have to worry about data loss. Plus I can time-travel files at hourly intervals.

One concern that occurred to me was the backup sits on my local hard drive. If the hard drive died, I would lose all my backups.

To solve that problem, I will have to go through the rabbit hole of doing backup for backups, or backup to a remote location or cloud, or setup a Raspberry Pi RAID.

At the moment, I'm not very concerned - I have Apple iCloud to back up my photos, videos, notes etc and I use GitHub to host my org-files and code. So having a backup for backups is not necessary for me for now.

I live in Emacs entirely apart from using browser for googling. Having an Emacs server running on the background makes Emacs available all the time. So I won't worry about closing it accidental.

It is not hard to do that, just run

emacs --daemon

in command line to start the Emacs server. It will load user configuration file as usual. Then run

emacsclient -c & 

to open an Emacs GUI instance that uses the Emacs server. That's how I have been doing for a while.

An better approach is using systemd. It is the services manager of Linux. Whenever my Debian 11 laptop boot up, systemd would start a bunch of services in parallel, for example, Networking manager connects WIFI, Bluetooth connects wireless keyboard so everything would be ready after I login. And I want Emacs to be ready as well.

I can achieve that by simply having an shell script automatically running after login. But there are benefits of using systemd. It has bunch of sub-commands for managing services, for example, checking logs, status etc.

It's a nice tool to have, I can use it for example Jupyter Notebook server.

That's why I pulled the trigger and spent 2 hours in implementing and testing it. Here's the technical bit.

In order to use systemd to manage Emacs server, I firstly need a configuration file (which is called unit file). Debian Wiki provides a short description of the syntax and parameter of unit file.

I found an simple one in Emacs Wiki. It looks like this

[Unit]
Description=Emacs text editor
Documentation=info:emacs man:emacs(1) https://gnu.org/software/emacs/

[Service]
Type=forking
ExecStart=/usr/bin/emacs --daemon
ExecStop=/usr/bin/emacsclient --eval "(kill-emacs)"
Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket
Restart=on-failure

[Install]
WantedBy=default.target

The important parameters are

ExecStart

It tells systemd what to do when starting Emacs

service, in this case it runs /usr/bin/emacs --daemon command.

ExecStop

it tells systemd what to do when shutting down Emacs

service, in this case it runs /usr/bin/emacsclient --eval "(kill-emacs)" command.

If you are using an Emacs built in a difference directory, you have to change /usr/bin/emacs to wherever your Emacs is located.

Then save the configuration file as ~.config/systemd/user/emacs.service/.

After that run

systemctl enable --user emacs

so systemd would copy the configuration file into central places and it would start Emacs service at boot time.

To run Emacs service right now, use

systemctl start --user emacs

This is what I see in my console

emacs.service - Emacs text editor
Loaded: loaded (/home/yitang/.config/systemd/user/emacs.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-06-14 09:12:26 BST; 24h ago
Docs: info:emacs
man:emacs(1)
https://gnu.org/software/emacs/
Main PID: 5222 (emacs)
Tasks: 5 (limit: 19027)
Memory: 154.7M
CPU: 3min 25.049s
CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/emacs.service
├─ 5222 /usr/bin/emacs --daemon
└─16086 /usr/bin/aspell -a -m -d en_GB -p /home/yitang/git/.emacs.d/local/ispell-dict --encoding=utf-8

Jun 14 09:11:57 7270 emacs[5222]: No event to add
Jun 14 09:11:57 7270 emacs[5222]: Package dash-functional is obsolete; use dash 2.18.0 instead
Jun 14 09:12:01 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/config/org-mode.el (source)...done
Jun 14 09:12:01 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/config/refile.el (source)...
Jun 14 09:12:01 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/config/refile.el (source)...done
Jun 14 09:12:01 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/config/scripting.el (source)...
Jun 14 09:12:26 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/config/scripting.el (source)...done
Jun 14 09:12:26 7270 emacs[5222]: Loading /home/yitang/git/.emacs.d/load_config.el (source)...done
Jun 14 09:12:26 7270 emacs[5222]: Starting Emacs daemon.
Jun 14 09:12:26 7270 systemd[4589]: Started Emacs text editor.

For far, I have the following two tweaks to make the usage of systemd more pleasant.

The systemd services under my account would only start after I login. Because I have tons of Emacs configuration, I still have to wait few seconds before Emacs server is ready. So it would be awesome to have the Emacs server starting to load before I login.

This doesn't seems to be simple to implement, because technically, it would require the Emacs server to be defined on system level, but it will load files in my personal home drive without me being logged in. It might be still okay since I'm the sole user of my laptop, but I have to tweak the permissions and would probably end up with non-secure permission setting.

So I leave this idea here.

As part of this team, I worked on final ensemble models. Immediately after join, i completed 5 tasks:

1. make sure everyone uses the same agreed cross validation schema. This is essential for building ensemble model.
2. provide model_zoo.md document to keep track of all level 1 models, their train/valid/lb scores, feature used, and file path to their oof/test prediction.
3. write merge_oof.py to combine all oof/test predictions together.
4. write R scripts for glmnet ensemble.
5. write python scripts for LightGBM ensemble.

Once new model is built, other team member update the model_zoo.md and upload the data to a private github repo. Then I update the merge_oof.py to include new models' result, and run glmnet and LightGBM ensemble. We had this ensemble workflow automated so it takes little effort to see the ensemble model's performance.

I spent some times analysing the coefficients/weights of L1 model and tried to exclude models with negative and lower weights. To my surprise it doesn't help at all. The final submission is a glmnet ensemble with 41 models (lgb + xgb + NN).

Also, LightGBM ensemble has much better cv score but the LB score is worse. I suspect it is because there are leakage in L1 models and glmnet is more robust to leakage since it's linear model. Unfortunately, there's no enough time to identify which models have leakage.

This is my 2nd time work in a team, although there's a lot space for improvement collaborating when compared with a professional data scientist team but as night/weekend project, we have done a really good job as a team.

The setup for collaboration:

1. Slack for discussion. we have channel for general, final_ensemble, random for cat photos etc.
2. we also used Slack for sharing features which i personal don't like.
3. Private github repo for sharing code and oof/test predictions.
4. Monday.com for managing tasks. it gives a nice overview of what everyone's up to.

we tried very hard to get a gold, but other teams work even more harder. At one point we were at 17, and finished at 22.

Finally, when we waited 1 hour for the final deadline, we had a lovely discussion about our past disqualification experience. We were all shocked when we were at different team in Toxic competition but team up with the same person. We shared their person's multiple Kaggle accounts and added to our personal block-list.

Before going into the technical details, you have to check that you are able to send out emails. You need the

• [ ] SMTP server,
• [ ] user name,
• [ ] password,
• [ ] port number, and
• [ ] communication protocol.

You could easily found out these information from your Email service provider. An example of Gmail is at Here.

To check if you have all the information correct, run the following snippet. It will try to send out an empty Email to yourself. Make sure you fill in the username and password before hit go.

  import smtplib
  username = <Fill In>
  password = <Fill In>
  conn = smtplib.SMTP("smtp.gmail.com")
  conn.starttls()  # set connection to TLS mode
  conn.login(username, password,)  # Log in to the remote server
  conn.sendmail(username, [username], 'For testing')  # Send emails
  conn.quit()  # close connection.
  

If you don't see error messages, that's great, you are all set. There should be an Email in your Inbox. It has no subject and for testing in the main body.

If you do, you need to double check the information, and try again. If you are sure that the information are correct, btu still can't send out, check your network configuration, maybe the firewall block the connection.

Once you are able to send out an empty email the next is to compose an full Email.

An Email is consisted of multiple parts, the Subject, Body, attachments, Signature, and also some meta-data including from, to, and date.

Firstly, create an object of MIMEMutiple() class. It will be the building block of your Email. The approached described here is to add each components into it.

At JBARML, we are planing to send users emails for a progress update. Many of the data generating process are lured together and automated by a workflow manager. The whole programcan takes upto weeks to complete. Actively sending update progeress to user is much more senssible then user loggin in to a remote machien and check it now and then.

In this case,

email to notify user the progress of the data generating workflow.