To do this you need two empty drives, with the guide below all data on the drive will be lost if it is not empty!

1. Install cryptsetup

apt-get install cryptsetup

2. Create the dm-crypt

cryptsetup -y -v luksFormat /dev/sdb
cryptsetup -y -v luksFormat /dev/sdc

(Make sure to use the correct device. To show available devices use „lsblk“.)

3. Mount the cryptsetup disk

cryptsetup luksOpen /dev/sdb disk1
cryptsetup luksOpen /dev/sdc disk2

4. Check if this was successful

ls -l /dev/mapper/disk*

or 
sudo cryptsetup -v status disk1
sudo cryptsetup -v status disk2

5. Create BTRFS volume RAID 1, this means all data and meta-data is mirrored, which means if you use 2 drives with i.e. 4TB each you can only use 4TG over all in the BTRFS mounted volume.

mkfs.btrfs -m raid1 -d raid1 /dev/mapper/disk1 /dev/mapper/disk2

You could list more than two drive, for example to increase redundancy, but if you want to use more drive to increase storage you would have to use a different Raid level.

6. Once you created a multi-device file system, you can use any device in the FS for the mount command

mount -t btrfs -o noatime,compress=zstd /dev/mapper/disk1 /mnt/btrfs

7. Creat a subvolume for each purpose/for each data storage folder you want to use the RAID 1 for

btrfs subvolume create /mnt/btrfs/subvolume1_data

For example if you want to store you /srv/http and your /var/lib/mysql on the BTRFS RAID drive you should create one subvolume for each, of them, i.e.

btrfs subvolume create /mnt/btrfs/subvolume_http
btrfs subvolume create /mnt/btrfs/subvolume_mysql

Those subvolumes can than be mounted in the respective folders. Before you mount them you should set the permissions/ownership of the folder as same as their mount folder (in this case check permissions of /srv/www and /var/lib/mysql), then move all files and folders from the current folder to the subvolume (see how to mount it below) and then mount the subvolume to the folder).

8. Mount the subvolume

mount -t btrfs -o subvolume=subvolume1_data,noatime,compress=zstd /dev/mapper/disk1 /mnt/data

This uses zstd compression for the mount, this is invoked only by setting the mount parameter. For details on compression see here.

You can mount this subvolume, also the main btrfs volume, to any folder you want, lets assume you want to mount it to /home/$USER/data, so instead of the last command you want to do this (if already mounted before, just do „umount /mnt/data“ if you already mounted it)

mount -t btrfs -o subvolume=subvolume1_data,noatime,compress=zstd /dev/mapper/disk1 /home/$USER/data

(folder must exists, „mkdir /home/$USER/data“ if you want to create it)

Now the subvolume is available in /home/$USER/data, see „df -h“ to check.

The output will contain a line like this:
/dev/mapper/disk1 3.7T 17M 3.7T 1% /home/YOUR_USER_NAME/data

9. Change permission of the subvolume

chown -R $USER:$USER ~/data

Now you can write and read from the folder, you have full access to it, which also means you have full access to the subvolume content and the permissions are inherited by folders and files you create in the folder „data“.

10. Show BTRFS filesystem

sudo btrfs filesystem show

The output will look like this

Label: none  uuid: 690fbca6-f764-4873-9f2c-f60c40197a14
        Total devices 2 FS bytes used 656.00KiB
        devid    1 size 3.64TiB used 2.03GiB path /dev/mapper/disk2
        devid    2 size 3.64TiB used 2.03GiB path /dev/mapper/disk1

There is a great cheat sheet for BTRFS here.

Please leave a comment if you have a suggestion.

I found something pretty useful, just a quick / code snippet to show the size of each table of the currently selected database.

SELECT table_name ,
  round(((data_length + index_length) / 1024 / 1024), 2) as SIZE_MB
FROM information_schema.TABLES
WHERE table_schema = DATABASE() ORDER BY SIZE_MB DESC;

You have to start ‚mysql‘ from CLI, in any terminal emulator of you choice, by adding the database name it should select when started, so lets say the database you’re looking to get the table sizes from is ‚wordpress‘ you’d have to call ‚mysql wordpress‘ and then in mysql you’d have to enter the whole code snippet from above and hit enter afterwards.

You’ll get a list of all tables ordered by their size, looking like this:

+---------------------------+---------+
| table_name                | SIZE_MB |
+---------------------------+---------+
| item                      | 1656.30 |
| item-body                 | 1166.86 |
| terms                     | 1029.72 |
| item-url                  |   11.89 |
| conv                      |    1.17 |
+---------------------------+---------+

Source

How to add a swap drive to your system, in this case I did this for a virtual machine (qemu) running Arch Linux.

See at the bottom for information about the virtual drive for qemu.

List current available to system disks:

sudo blkid

List all disks:

sudo lsblk

Format as swap:

sudo mkswap /dev/vdb

gives back the UUID

Now the new swap disk will be shown when executing this again:

sudo blkid

(also give the UUID)

Take the UUID from above and add an entry in /etc/fstab like this:

echo "UUID=56445300-90f5-40b4-860b-99037d0e8aad none swap sw 0 0" |sudo tee -a /etc/fstab

Actiate available swap disks, ie. from fstab or on GPT volume.

sudo swapon -a

Check the swap is available now:

sudo swapon --show

The qemu image I created as „raw“, VirtIO and set the cache mode to „writeback“, this should be fine for swap while being fast too. See explanation here and here.

I also looked at this information about how to add a swap disk.

As always, give me your feedback, I will add information that can help in the future.

… is an easy to use and powerful tool, it can be used to replace part of file name (string1) with some other string. Something one can also do with some „sed“ hack, but this tool is much easier to use, especially for beginners.

You can get all information needed from „rename –help“ (here from Linux Mint):

rename --help
Usage:
    rename [ -h|-m|-V ] [ -v ] [ -n ] [ -f ] [ -e|-E perlexpr]*|perlexpr
    [ files ]

Options:
    -v, -verbose
            Verbose: print names of files successfully renamed.

    -n, -nono
            No action: print names of files to be renamed, but don't rename.

    -f, -force
            Over write: allow existing files to be over-written.

    -h, -help
            Help: print SYNOPSIS and OPTIONS.

    -m, -man
            Manual: print manual page.

    -V, -version
            Version: show version number.

    -e      Expression: code to act on files name.

            May be repeated to build up code (like "perl -e"). If no -e, the
            first argument is used as code.

    -E      Statement: code to act on files name, as -e but terminated by
            ';'.

I did use this tool rename some files that contained the string „www“, which I had to replace with „web“, lets say there were files named like this:

www.example.com-97126.ccd
www.example.com-54852.ccd
www.example.com-87430.ccd
www.example.com-75413.ccd

For Ubuntu, maybe also Debian, based systems I use the following command to rename all above files:

rename -e 's/www/web/' *.ccd

After this the files are now called
web.example.com-….
….

The only problem is, the syntax varies from one Linux distribution to another, so check the syntax before you use rename.

For Arch Linux I found that something like this would work:

rename www web *.ccd

The expression „*.ccd“ defines which files to act on, so this will act on all files that match „*.ccd*, in the example above you could also use „www.example.com-*.ccd“ or something similar.

Main thing I wanted to get out there is, there is an easy tool to batch rename files instead of some complex if-for bash code. Which I tried to use before but failed at.

This will be a short post about the above mentioned parts, lets start right away.

SSH config

Every user can have a SSH client configuration file, it is located in ~/.ssh/config (where config ist the filename).
My SSH config file, I guess as many other users file, grew over time. Lets start with some things from my file.

There is the Host part, it usually contains the name you want to assign to the host to use it later on. Tipp, use some short and of course unique name.

Host myserver

The Host part can be wildcard, then the following parameters are valid for any Host. It then looks like this:

Host *
     ServerAliveInterval 15
     ControlMaster auto
     ControlPath /tmp/ssh-%r@%h:%p

ServerAliveInterval
Sets a timeout interval in seconds after which if no data has been received from the server, ssh(1) will send a message through the encrypted channel to request a response from the server. The default is 0, indicating that these messages will not be sent to the server. This option applies to protocol version 2 only.

This is copied from the man page of „ssh_config„.

In the case above it means the client will keep the connection alive by sending some packet every 15 seconds at least. The packet is send and the client will wait for a reply for 3 times the interval (2 packet can get lost) by default. I use this setting to be sure the connection is kept alive and it is still alive. I work on a mobile connection quite often.

ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

These two options might be helpful in particular cases when you use multiple SSH connection to one host and want to save some time to build up the connection. The option is explained here and the options „/tmp/ssh-%r@%h:%p“ are explained very well here. Basically it will use a socket to connect to the host and the socket will be in „/tmp/ssh-%r@%h:%p“, where „r“ is the remote user name, „h“ is the host name and „p“ is the port number. The use of all this ensures that the socket name is unique.

Host myserver
	HostName example.org
	User username
        Port 22
#	IdentityFile ~/.ssh/id_rsa
	ForwardAgent yes
	Compression yes
	#CompressionLevel 9

HostName contains the remote IP or host name (domain name) you want to connect to, like in „ssh username@example.org“ the example.org.

User contains the remote username.

This two are the minimum configuration parameters you want to use. When you want to connect to example.org via SSH you now just have to type the command „ssh myserver“ and it will connect you as you are used to, but with much less typing.

Port is not needed if you want to use 22 (standard SSH port), so it is useless in the example above, but if you want to connect to some host that runs the SSH service on some non standard port, lets say 6022 you can use „Port 6022“ to provide the port.

IdentityFile contains the identity file specific to the host. I commented this out some time ago because I now use a SSH-Agent, which you can do as well, there is „ssh-add“ on any standard Linux system. So when I start a session I just do „ssh-add .ssh/id_rsa .ssh/id2_rsa“ and so on to add all my needed ssh keys to the SSH-Agent.

With ForwardAgent your available keys from you local ssh-agent can be used on the remote host too, this is helpful when from the remote host you want to ssh to another server. I would suggest to use this option on any host, so you could move it to the „Host *“ section.
Using this parameter will be handy if you want to rsync from one server to another.

You can use Compression if you are using a slow connection of if you transfer big amounts of data via SSH, it can be a bottle neck if your CPUs (local and remote) are not that fast. So it may be better to use Compression only in special cases or on really slow connections.
With CompressionLevel you can set the compression level, it is like with most other compressions and I came to the conclusion to comment it out with the #, so it is not used in my configuration anymore. Check the man page of gzip or similar to see the compression levels explanation from 1-9.

So if you want you can use the „Host *“ section like this:

Host *
	ServerAliveInterval 15
	ControlMaster auto
	ControlPath /tmp/ssh-%r@%h:%p
    	ForwardAgent yes

Jump hosts

In some cases I want to connect to some host that is not reachable from the outside world (aka. the Internet), in this cases I use a jump host that is connected to that internal network.
I had to work some years the usual way, the usual way being to connect to the jump host by „ssh jumphost“ and then open another ssh connection from its local command line interface. Instead you can add the following to your ssh configuration for the host in the internal network.

ssh -q -W %h:%p myserver

Explanation from explainshell.com:

ssh(1) -q -W %h:%p myserver

OpenSSH SSH client (remote login program)
-q Quiet mode. Causes most warning and diagnostic messages to be suppressed.
-W host:port Requests that standard input and output on the client be forwarded to host on port over the secure channel. Implies -N, -T, ExitOnForwardFailure and ClearAllForwardings and works with Protocol version 2 only.
ssh connects and logs into the specified hostname (with optional user name). The user must prove his/her identity to the remote machine using one of several methods depending on the protocol version used (see below). If command is specified, it is executed on the remote host instead of a login shell.

So when you add that to another Host configuration like this:

Host machine2
        HostName 192.168.0.42
        User username
        ssh -q -W %h:%p myserver

This would mean that your ssh connection to 192.168.0.42 as user username would be made via myserver and all data would be forwarded via myserver. This can also be used to connect via a particular server all the time if it is added to the „Host *“ section of you ssh config.

SSH config for hidden Tor service SSH server

Something else I like to use ssh config for, is to connect to SSH via a tor Onion service, which I like to use for headless small computers (portable computer without a screen) like a Raspberry Pi that I connect to some network and want to SSH into them without knowing their IP address in the LAN. The setup of a Tor hidden service can be found online, the basic way it works is that your raspberry pi will tunnel to the Internet and be reachable via a hidden service.

Host raspberrypi
	HostName ap37csfkkmkksmkktllhjkcqd.onion 
        User Pi
        Port 22
        VerifyHostKeyDNS no
        ProxyCommand /bin/nc -xlocalhost:9050 -X5 %h %p

HostName now contains the .Onion domain of the hidden service.

I use VerifyHostKeyDNS to avoid warning about changed DNS for that host. I am not sure if it is actually needed, but at some point I added it.

The ProxyCommand parameter now is the important thing. I requires you to have Tor running as a client, this should be possible by „sudo apt-get install tor && sudo systemctl enable tor.service –now“ Probably have to run it in two commands.
This uses netcat (nc) to forward all packets from ssh to nc to the local port 9050 (standard Tor Socks port).
I never noted what exactly the parameters for nc do. 🙂 if anyone know please let me know. I tried a quick search but it did not bring up any helpful results.

Port must contain the port the hidden service ssh is available at, this can be different from the clients ssh daemon/server but is defined by the hidden service configuration in the torrc config file.

Have a look in the manpage of ssh by typing „man ssh“ in your command line or open the manpage online. The ssh_config manpage is also very helpful.

autossh

I lately started to use autossh in combination with remote screen sessions, screen is the tool to use if you work remotely and do something that you want to continue after a possible connection interruption. There is tmux as well, but I prefer screen. Please have a look at both and choose which you prefer and use it.

Quote from the manpage of autossh:

autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic.

It basically improves the interruption resistivity of ssh by wrapping some other things around the connection and by using some special parameters. As I said I work remotely often and also I use mobile connections often on trains as well, I uses mosh (MObile SHell) before, but it is not working with screen that well, also it had some other downsides and to I ended up with autossh.

Let me know if there is any faults in this summary about my ssh use and setup.

Edit:
I figured something else out, I always wanted to keep my ssh config in sync between some computers. The problem was the synced folder is not ~/.ssh but ~/some/other/folder and in there I have a „ssh_config“ file. So I tried to symlink that to my ~/.ssh which did not work (ln -s ~/some/other/folder/ssh_config ~/.ssh/config did not work) because auf some permission issue I think. Host in the file were just not found.
Today I figured out that I can include another file in my ~/.ssh/config by adding the following line in my ssh config file:

Include ~/some/other/folder/ssh_config

This now allows me to sync ~/some/other/folder/ssh_config between computers and include that file in my ssh config files on any computer I like to.

VM runterfahren

Image der VM vergrössern:

sudo qemu-img resize /var/lib/libvirt/images/vm_disk.img +15GB

Je nach Setup ist sudo nicht nötig, durch den Befehl wird die Diskimagedatei um 15 GB vergrößert.

VM starten

Weiter geht es in der VM auf der Konsole.

Partitioen mit parted anzeigen lassen:

parted -l

Ausgabe so oder ähnlich:
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 53,7GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
1 1049kB 53,7GB 53,7GB primary btrfs boot

Ich habe, bevor ich angefangen habe, die VM heruntergefahren und eine Kopie des Diskimage erstellt. Deswegen habe ich mich dann entschieden die Partition in der laufenden VM anzupassen, grundsätzlich sollte das bei Linux gehen. Auflösung, es hat geklappt. Also weiter.

Disk auswählen bei Start von Parted:

sudo parted /dev/vda

In parted the Partitionen nochmal anzeigen:

print

Ausgabe so oder ähnlich:
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 69,8GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
1 1049kB 53,7GB 53,7GB primary btrfs boot

In parted die Partition (1) vergrößern:

resizepart 1

Dann die Abfragen bestätigen, maximale Größe eingeben (max oder 100% scheinen nicht mehr zu funktionieren) und bestätigen das der Vorgang durchgeführt werden soll trotz das die Partition gemountet ist.

Zu guter Letzt noch Dateisystem (btrfs) vergrössern:

sudo btrfs filesystem resize max /

Anmerkung: Bei anderen Dateisystemen ist der Befehl ein anderer. Im Beispiel ist der Mountpoint „/“, ggf. muss die auch angepasst werden.

It drives me nuts that I cannot copy something from a ssh terminal when I use vim. I have no clue what this visual mode is meant to do besides anoying my.

My solution:

echo "set mouse-=a" >> ~/.vimrc 

no more visual mode -> marking and copying text from vim by using the mouse works again.

A short one:

Command

lsof -i -P -n | grep LISTEN

This will output all the ports the system is listing on. You might have to prepend „sudo“.

So then the command will be:

sudo lsof -i -P -n | grep LISTEN

The list is long without the „grep“, it will filter the output and only show the lines which contain „LISTEN“.

Edit:
Oliver from Hubzilla (part of the Fediverse) pointed out that

netstat -anpl

might be helpful too. The command shows all the listing sockets of the Linux machine it is run on. (I added the „l“ parameter to just show listing sockets.

This is kind of my cheat sheet, nothing fancy, just what I need.

Command:

rsync -chavzP --stats 'host:~/RemotePath' ./LocalPath

rsync – Befehl
-c, –checksum
-h, –human-readable
-a, –archive
-v, –verbose
-z, –compress
-P – equivalent to –partial –progress
–stats – This tells rsync to print a verbose set of statistics on the file transfer
‚host:~/RemotePath‘ – is the source path (in this case a remote path.
./LocalPath – ist the destination

Some more variants:

from local to remote

rsync -chavzP --stats ~/FileName.file 'host:~/path/'

Leave the z if compression does not matter (in most cases it does not matter).

rsync -chavP --stats ~/FileName.file 'host:~/path/'

Rsync to MOVE a folder from A to B. Source will be deleted!!

rsync -hcavP --stats --remove-source-files ./source ./destination_folder

Rsync can be used from local to local to copy files. It can be used from ‚remote‘ to local or vom local to ‚remote‘, unfortunately it can not copy from remote to remote.

Don’t forget to include the ticks around the remote (host) part.

I this „rsync -chavzP –stats ~/FileName.file ‚host:~/path/'“ the „host“ is defined in my ~/.ssh/config, but you can also do something like this:
„rsync -chavzP –stats ~/FileName.file ‚user@example.org:~/path/'“

This even work via ssh jump hosts. I will write something about that soon.