Migrating emails from one server to another using imap

I recently switched the email hosting from one provider to another. I always set email up as IMAP so the mail is stored and can be accessed and synchronised on multiple devices. This ‘How to’ is based on you having a similar setup. There are tools out there that perform this task automatically but being a control freak I wanted to do this manually to ensure all went well.

Presumptions:

  • You have an IMAP mail account on one server, and you want to migrate this account and the messages to another server.
  • The old server is still up and running and the nameservers still point to the existing hosting (e.g. the server you are migrating the messages from)
  1. Create a mail account with the new provider that exactly mirrors settings of the existing account on the old provider e.g. same user, same password, same email address
  2. Install an additional email client on your desktop system – I recommend a separate client from your main client in case it all goes awry. I’ve successfully tested it with the same client when using Thunderbird 3 but ymmv.
  3. Set up the old mail account in thunderbird  (the provider you’re migrating from)
  4. Important: Ensure the account is setup as IMAP in Thunderbird
  5. Deselect any default junk mail options (This stops Thunderbird  from automatically moving mails to different mail boxes)
  6. Let Thunderbird connect to the old mail account and download all emails
  7. Set up the new mail account in thunderbird (the provider you’re migrating to) – You will need to use the ip address of the new server as the imap and smtp dns names aren’t pointing to the new provider yet
  8. Once Thunderbirds has connected and and set up the inbox you will need to create the folder in the new mail account, mirroring the folders in the old account on the old servers
  9. In the OLD Mail account (the one you are migrating from), select each mail folder, CTRL+A to select all the messages, right click and choose ‘Copy to…’ and select the equivalent folder on the NEW server version of the account
  10. Repeat step 9 for all folders/messages in the account (don’t forget the Sent items!)

Upgrading Debian 8 ‘Jessie’ to Debian 9 ‘Stretch’

Intro

Debian 9 Stretch was released as the latest stable version of the Linux Distribution: Debian. While it’s always possible to install Debian 9 fresh from scratch, it’s also possible to perform an in-place upgrade from Debian 8. The following post describes the necessary steps to do so.

For a incredibly thorough documentation of the process, I suggest you also read through the official release notes.

Notes:

  • Upgrading to Debian 9 Stretch is only supported from Debian 8 Jessie. If you are running a version older than 8, you must first upgrade to 8 before working through this process.
  • The upgrade involves a kernel update, so a reboot will be required toward the end of the process.
  • It is strongly recommended that you have a full system backup or backup of any important data before proceeding with the upgrade, ensure that you have a plan to roll back. In the case of a virtual machine, take a snapshot before starting.

Performing the upgrade to Debian 9 Stretch

Before proceeding with the upgrade, please read through the list of issues to be aware of when upgrading to Stretch.

  1. It is recommended that you have your Debian 8 Jessie installation completely up to date before starting, to do this run “apt-get update” followed by “apt-get upgrade” and install available updates.
root@debian8:~# apt-get update
root@debian8:~# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

In this case all updates have been applied already, so it’s fine to proceed.

Edit the /etc/apt/sources.list file, my file is shown below. As you can see all of the lines are currently specifying “jessie”. Note that your mirror sources will likely be different which is fine.

deb http://ftp.ch.debian.org/debian/ jessie main
deb-src http://ftp.ch.debian.org/debian/ jessie main

deb http://security.debian.org/ jessie/updates main contrib
deb-src http://security.debian.org/ jessie/updates main contrib

# jessie-updates, previously known as 'volatile'
deb http://ftp.ch.debian.org/debian/ jessie-updates main contrib
deb-src http://ftp.ch.debian.org/debian/ jessie-updates main contrib

Change the instances of “jessie” to “stretch”, you can either do this manually, or automatically with the below sed command.

sed -i 's/jessie/stretch/g' /etc/apt/sources.list

You can either use “stretch” or “stable”, as Debian 9 Stretch is now the current stable version as of writing. However note that if you use stable instead of the specific release name, in future when Debian 10 is released that will be the stable version so you may upgrade to that unintentionally.

The recommended way to upgrade Debian is with the ‘apt-get’ command. First update the list of available packages with the below command, as we’ve just updated the sources.list file.

apt-get update

Use “apt list –upgradable” command to quickly see what will be installed, updated, and removed during the upgrade process without affecting the system.

apt list --upgradable

Sample Output:

unattended-upgrades/stable 0.93.1+nmu1 all [upgradable from:0.83.3.2+deb8u1]
util-linux/stable 2.29.2-1 amd64 [upgradable from: 2.25.2-6]
uuid-runtime/stable 2.29.2-1 amd64 [upgradable from: 2.25.2-6]
vim/stable 2:8.0.0197-4 amd64 [upgradable from: 2:7.4.488-7+deb8u3]
vim-common/stable 2:8.0.0197-4 amd64 [upgradable from: 2:7.4.488-7+deb8u3]
vim-runtime/stable 2:8.0.0197-4 all [upgradable from: 2:7.4.488-7+deb8u3]
vim-tiny/stable 2:8.0.0197-4 amd64 [upgradable from: 2:7.4.488-7+deb8u3]

Now that the list of available packages has been updated from the mirror, run the below command to perform a minimal upgrade.

apt-get upgrade

This is known as a minimal system upgrade as it only upgrades packages that can be upgraded without needing any other packages to be removed or installed, so it’s a safe place to start. This upgraded 932 packages requiring 412MB on my system.

Now you’re ready to do the complete system upgrade, this will upgrade to the latest available version for all packages installed.

apt-get dist-upgrade

Ensure that you have enough free disk space to complete the operation, in my case it notes that afterwards 1,048MB of additional disk space will be used with 639 package upgrades and 479 newly installed packages.

During the distribution upgrade, services installed on your system needs to be restarted after up gradation of each service packages (ex. Apache, NTP) which may cause you the service interruptions. You can choose to restart automatically during upgrade or manually after the upgrade. Here, I opted to do “an automatic restart of services during the OS upgrade“.

Verify upgrade:

Reboot your machine after the distribution upgrade.

reboot

Verify the current version of Debian operating system.

lsb_release -a

Distributor ID: Debian
Description: Debian GNU/Linux 9.0 (stretch)
Release: 9.0
Codename: stretch

Users susceptible to man-in-the-middle attacks due to corporate https inspection

A large number of companies use “security” products to inspect HTTPS traffic for detecting malware and prevent other types of attacks. However, they might inadvertently make their user’s more susceptible to man-in-the-middle attacks by  decrypting and re-encrypting HTTPS connections.

The U.S. Computer Emergency Readiness Team (US-CERT) warns in an advisory that HTTPS inspection products don’t mirror the security attributes of the original HTTPS connections between the client and the server (Mirror: HTTPS Interception Weakens TLS Security | US-CERT).

HTTPS inspection is deployed in companies for checking the encrypted traffic coming from an HTTPS website to make sure it does not contain any malware or any other type of attacks. It basically performs a decryption and re-encryption of the client’s connection to an HTTPS server. The “security” products (proxy, web-gateway, firewall etc.) establish the connection on the client’s behalf by first decrypting the client’s HTTPS connection and re-encrypting the traffic sent to the HTTPS server. The client is served with a different, locally generated certificate by the security product which essentially perform a man-in-the-middle attack.

In some enterprise environments, an HTTPS connection may even be intercepted and re-encrypted multiple times. For example, at the network perimeter by a security gateway product and later, on the endpoint by a client’s antivirus program which needs to inspect the traffic for malware.

The problem revolves around the fact that the client’s browser no longer validates the real certificate issued by the server because its replaced with a locally generated certificate from the security product. In return, the task of validating the certificate now falls to the intercepting proxy.

According to the published advisory, those security products are evidently pretty bad at validating server certificates. An investigation conducted by researches from Google, Mozilla, Cloudfare, and multiple Universities states that the intercepted connections use weaker cryptographic algorithms (Source: interception-ndss17). The security products even advertise support for known-broken encryption ciphers that would allow an active man-in-the-middle attack by intercepting and downgrading a connection in order to decrypt it.

The analysis by the researches found that at least 32 percent of connections to e-comerce sites and 54 percent of Cloudflare HTTPS connections, which were intercepted, became less secure than they would have been if the user had connected directly to the server.

Browser makers had a long time to properly unterstand the quirks of TLS connections and certificate validation. Therefore, there is no better client-side implementation of TLS, the protocol used for encrypting HTTPS connection, than the one found in modern browsers.
In comparison, security product vendors use outdated, customised TLS libraries where they even back-port new protocol features. Re-implementing those features found in newer libraries makes them susceptible to serious vulnerabilities.

Furthermore, the US-CERT points out another widespread problem that many products intercepting HTTPS don’t properly validate the certificate chain presented by servers. Certificate-chain verification errors are infrequently forwarded to the client, leading the client to believe that operations were performed with the correct server.

The BadSSL website allows organisations and even employees to check if their HTTPS inspection products improperly validate certificates or allow for insecure ciphers. The client test from Qualys SSL Labs also provides checks for some known TLS vulnerabitiles and weakenesses.

 

Homebrew on macOS 10.12 Sierra

Apple introduced a few changes with the directory structure and permissions in macOS Sierra 10.12 which in the beginning broke Homebrew on macOS without some CLI magic to manually fix things. However, this has been fixed and Homebrew now full supports macOS 10.12. One of the issues was around a change in permissions on the directory /usr/local which on a new installation either doesn’t exist or wasn’t writeable.

Therefore Homebrew changed their installation routine and migrated the installation folder to /usr/local/homebrew in order to circumvent the whole permissions issues.

Install Homebrew

The following Terminal command will download and install Homebrew. Ruby comes pre-installed with macOS and therefore just copy and paste the following command in your Terminal:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Homebrew requires the command line tools which ship with Xcode. If you don’t have Xcode or the command line tools installed, macOS will automatically prompt you to install them. You can get away with the command line tools by clicking install.

However, you can also get Xcode by downloading it free of charge from the AppStore. Here, I will simply install the command line tools.

To check for any issues after the installation, run:

brew doctor

To search for an application:

brew search <application_name>

To install an application:

brew install <application_name>

To list all applications installed by Homebrew:

brew list

To remove an installation application:

brew remove <application_name>

To update Homebrew:

brew update

Fixing brew after upgrade to macOS 10.12

If you had brew installed before upgrading to macOS 10.12 you will notice that the upgrade broke brew. The following instructions will fix the broken install and will let you upgrade brew to the latest and greatest which supports 10.12

Accept the license agreement of the command line tools shipped with Xcode

sudo xcodebuild -license

Change ownership of /usr/local to yourself:

sudo chown -R $(whoami) /usr/local

Run brew doctor and brew update. The installer of the update will inform you that it will migrate the brew repository to a new directory:

brew doctor && brew update

=> Migrating HOMEBREW_REPOSITORY (please wait)...
==> Migrated HOMEBREW_REPOSITORY to /usr/local/Homebrew!
Homebrew no longer needs to have ownership of /usr/local. If you wish you can return /usr/local to its default ownership with:
sudo chown root:wheel /usr/local

Change the owner back to the defaults with:

sudo chown root:wheel /usr/local