• To do this you need two empty drives, with the guide below all data on the drive will be lost if it is not empty!

    1. Install cryptsetup

    apt-get install cryptsetup

    2. Create the dm-crypt 

    cryptsetup -y -v luksFormat /dev/sdb
cryptsetup -y -v luksFormat /dev/sdc

    (Make sure to use the correct device. To show available devices use „lsblk“.)

    3. Mount the cryptsetup disk

    cryptsetup luksOpen /dev/sdb disk1
cryptsetup luksOpen /dev/sdc disk2

    4. Check if this was successful

    ls -l /dev/mapper/disk*

or 
sudo cryptsetup -v status disk1
sudo cryptsetup -v status disk2

    5. Create BTRFS volume RAID 1, this means all data and meta-data is mirrored, which means if you use 2 drives with i.e. 4TB each you can only use 4TG over all in the BTRFS mounted volume.

    mkfs.btrfs -m raid1 -d raid1 /dev/mapper/disk1 /dev/mapper/disk2

    You could list more than two drive, for example to increase redundancy, but if you want to use more drive to increase storage you would have to use a different Raid level.

    6. Once you created a multi-device file system, you can use any device in the FS for the mount command

    mount -t btrfs -o noatime,compress=zstd /dev/mapper/disk1 /mnt/btrfs

    7. Creat a subvolume for each purpose/for each data storage folder you want to use the RAID 1 for

    btrfs subvolume create /mnt/btrfs/subvolume1_data

    For example if you want to store you /srv/http and your /var/lib/mysql on the BTRFS RAID drive you should create one subvolume for each, of them, i.e.

    btrfs subvolume create /mnt/btrfs/subvolume_http
btrfs subvolume create /mnt/btrfs/subvolume_mysql

    Those subvolumes can than be mounted in the respective folders. Before you mount them you should set the permissions/ownership of the folder as same as their mount folder (in this case check permissions of /srv/www and /var/lib/mysql), then move all files and folders from the current folder to the subvolume (see how to mount it below) and then mount the subvolume to the folder).

    8. Mount the subvolume

    mount -t btrfs -o subvolume=subvolume1_data,noatime,compress=zstd /dev/mapper/disk1 /mnt/data

    This uses zstd compression for the mount, this is invoked only by setting the mount parameter. For details on compression see here.

    You can mount this subvolume, also the main btrfs volume, to any folder you want, lets assume you want to mount it to /home/$USER/data, so instead of the last command you want to do this (if already mounted before, just do „umount /mnt/data“ if you already mounted it)

    mount -t btrfs -o subvolume=subvolume1_data,noatime,compress=zstd /dev/mapper/disk1 /home/$USER/data

    (folder must exists, „mkdir /home/$USER/data“ if you want to create it)

    Now the subvolume is available in /home/$USER/data, see „df -h“ to check.

    The output will contain a line like this:
    /dev/mapper/disk1 3.7T 17M 3.7T 1% /home/YOUR_USER_NAME/data

    9. Change permission of the subvolume

    chown -R $USER:$USER ~/data

    Now you can write and read from the folder, you have full access to it, which also means you have full access to the subvolume content and the permissions are inherited by folders and files you create in the folder „data“.

    10. Show BTRFS filesystem

    sudo btrfs filesystem show

    The output will look like this

    Label: none  uuid: 690fbca6-f764-4873-9f2c-f60c40197a14
        Total devices 2 FS bytes used 656.00KiB
        devid    1 size 3.64TiB used 2.03GiB path /dev/mapper/disk2
        devid    2 size 3.64TiB used 2.03GiB path /dev/mapper/disk1

    There is a great cheat sheet for BTRFS here.

    Please leave a comment if you have a suggestion.

    Keine Kommentare zu Create Linux dm-crypt BTRFS RAID 1

  • ,

    I found something pretty useful, just a quick / code snippet to show the size of each table of the currently selected database. 

    SELECT table_name ,
  round(((data_length + index_length) / 1024 / 1024), 2) as SIZE_MB
FROM information_schema.TABLES
WHERE table_schema = DATABASE() ORDER BY SIZE_MB DESC;

    You have to start ‚mysql‘ from CLI, in any terminal emulator of you choice, by adding the database name it should select when started, so lets say the database you’re looking to get the table sizes from is ‚wordpress‘ you’d have to call ‚mysql wordpress‘ and then in mysql you’d have to enter the whole code snippet from above and hit enter afterwards.

    You’ll get a list of all tables ordered by their size, looking like this:

    +---------------------------+---------+
| table_name                | SIZE_MB |
+---------------------------+---------+
| item                      | 1656.30 |
| item-body                 | 1166.86 |
| terms                     | 1029.72 |
| item-url                  |   11.89 |
| conv                      |    1.17 |
+---------------------------+---------+

    Source

    Keine Kommentare zu Size of each table in a database of MySQL or MariaDB

  • ,

    How to add a swap drive to your system, in this case I did this for a virtual machine (qemu) running Arch Linux.

    See at the bottom for information about the virtual drive for qemu.

    List current available to system disks:

    sudo blkid

    List all disks:

    sudo lsblk

    Format as swap:

    sudo mkswap /dev/vdb

    gives back the UUID

    Now the new swap disk will be shown when executing this again:

    sudo blkid

    (also give the UUID)

    Take the UUID from above and add an entry in /etc/fstab like this:

    echo "UUID=56445300-90f5-40b4-860b-99037d0e8aad none swap sw 0 0" |sudo tee -a /etc/fstab

    Actiate available swap disks, ie. from fstab or on GPT volume. 

    sudo swapon -a

    Check the swap is available now:

    sudo swapon --show

    The qemu image I created as „raw“, VirtIO and set the cache mode to „writeback“, this should be fine for swap while being fast too. See explanation here and here.

    I also looked at this information about how to add a swap disk.

    As always, give me your feedback, I will add information that can help in the future.

    Keine Kommentare zu How to add a swap drive to your system

  • … is an easy to use and powerful tool, it can be used to replace part of file name (string1) with some other string. Something one can also do with some „sed“ hack, but this tool is much easier to use, especially for beginners.

    You can get all information needed from „rename –help“ (here from Linux Mint):

    rename --help
Usage:
    rename [ -h|-m|-V ] [ -v ] [ -n ] [ -f ] [ -e|-E perlexpr]*|perlexpr
    [ files ]

Options:
    -v, -verbose
            Verbose: print names of files successfully renamed.

    -n, -nono
            No action: print names of files to be renamed, but don't rename.

    -f, -force
            Over write: allow existing files to be over-written.

    -h, -help
            Help: print SYNOPSIS and OPTIONS.

    -m, -man
            Manual: print manual page.

    -V, -version
            Version: show version number.

    -e      Expression: code to act on files name.

            May be repeated to build up code (like "perl -e"). If no -e, the
            first argument is used as code.

    -E      Statement: code to act on files name, as -e but terminated by
            ';'.

    I did use this tool rename some files that contained the string „www“, which I had to replace with „web“, lets say there were files named like this:

    www.example.com-97126.ccd
    www.example.com-54852.ccd
    www.example.com-87430.ccd
    www.example.com-75413.ccd

    For Ubuntu, maybe also Debian, based systems I use the following command to rename all above files:

    rename -e 's/www/web/' *.ccd

    After this the files are now called
    web.example.com-….
    ….

    The only problem is, the syntax varies from one Linux distribution to another, so check the syntax before you use rename.

    For Arch Linux I found that something like this would work:

    rename www web *.ccd

    The expression „*.ccd“ defines which files to act on, so this will act on all files that match „*.ccd*, in the example above you could also use „www.example.com-*.ccd“ or something similar.

    Main thing I wanted to get out there is, there is an easy tool to batch rename files instead of some complex if-for bash code. Which I tried to use before but failed at.

    Keine Kommentare zu rename

  • This will be a short post about the above mentioned parts, lets start right away.

    SSH config

    Every user can have a SSH client configuration file, it is located in ~/.ssh/config (where config ist the filename).
    My SSH config file, I guess as many other users file, grew over time. Lets start with some things from my file.

    There is the Host part, it usually contains the name you want to assign to the host to use it later on. Tipp, use some short and of course unique name. 

    Host myserver

    The Host part can be wildcard, then the following parameters are valid for any Host. It then looks like this: 

    Host *
     ServerAliveInterval 15
     ControlMaster auto
     ControlPath /tmp/ssh-%r@%h:%p

    ServerAliveInterval
    Sets a timeout interval in seconds after which if no data has been received from the server, ssh(1) will send a message through the encrypted channel to request a response from the server. The default is 0, indicating that these messages will not be sent to the server. This option applies to protocol version 2 only.

    This is copied from the man page of „ssh_config„.

    In the case above it means the client will keep the connection alive by sending some packet every 15 seconds at least. The packet is send and the client will wait for a reply for 3 times the interval (2 packet can get lost) by default. I use this setting to be sure the connection is kept alive and it is still alive. I work on a mobile connection quite often. 

    ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

    These two options might be helpful in particular cases when you use multiple SSH connection to one host and want to save some time to build up the connection. The option is explained here and the options „/tmp/ssh-%r@%h:%p“ are explained very well here. Basically it will use a socket to connect to the host and the socket will be in „/tmp/ssh-%r@%h:%p“, where „r“ is the remote user name, „h“ is the host name and „p“ is the port number. The use of all this ensures that the socket name is unique. 

    Host myserver
	HostName example.org
	User username
        Port 22
#	IdentityFile ~/.ssh/id_rsa
	ForwardAgent yes
	Compression yes
	#CompressionLevel 9

    HostName contains the remote IP or host name (domain name) you want to connect to, like in „ssh username@example.org“ the example.org.

    User contains the remote username.

    This two are the minimum configuration parameters you want to use. When you want to connect to example.org via SSH you now just have to type the command „ssh myserver“ and it will connect you as you are used to, but with much less typing.

    Port is not needed if you want to use 22 (standard SSH port), so it is useless in the example above, but if you want to connect to some host that runs the SSH service on some non standard port, lets say 6022 you can use „Port 6022“ to provide the port.

    IdentityFile contains the identity file specific to the host. I commented this out some time ago because I now use a SSH-Agent, which you can do as well, there is „ssh-add“ on any standard Linux system. So when I start a session I just do „ssh-add .ssh/id_rsa .ssh/id2_rsa“ and so on to add all my needed ssh keys to the SSH-Agent.

    With ForwardAgent your available keys from you local ssh-agent can be used on the remote host too, this is helpful when from the remote host you want to ssh to another server. I would suggest to use this option on any host, so you could move it to the „Host *“ section.
    Using this parameter will be handy if you want to rsync from one server to another.

    You can use Compression if you are using a slow connection of if you transfer big amounts of data via SSH, it can be a bottle neck if your CPUs (local and remote) are not that fast. So it may be better to use Compression only in special cases or on really slow connections.
    With CompressionLevel you can set the compression level, it is like with most other compressions and I came to the conclusion to comment it out with the #, so it is not used in my configuration anymore. Check the man page of gzip or similar to see the compression levels explanation from 1-9.

    So if you want you can use the „Host *“ section like this:

    Host *
	ServerAliveInterval 15
	ControlMaster auto
	ControlPath /tmp/ssh-%r@%h:%p
    	ForwardAgent yes

    Jump hosts

    In some cases I want to connect to some host that is not reachable from the outside world (aka. the Internet), in this cases I use a jump host that is connected to that internal network.
    I had to work some years the usual way, the usual way being to connect to the jump host by „ssh jumphost“ and then open another ssh connection from its local command line interface. Instead you can add the following to your ssh configuration for the host in the internal network.

    ssh -q -W %h:%p myserver

    Explanation from explainshell.com:

    ssh(1) -q -W %h:%p myserver

    OpenSSH SSH client (remote login program)
    -q Quiet mode. Causes most warning and diagnostic messages to be suppressed.
    -W host:port Requests that standard input and output on the client be forwarded to host on port over the secure channel. Implies -N, -T, ExitOnForwardFailure and ClearAllForwardings and works with Protocol version 2 only.
    ssh connects and logs into the specified hostname (with optional user name). The user must prove his/her identity to the remote machine using one of several methods depending on the protocol version used (see below). If command is specified, it is executed on the remote host instead of a login shell.

    So when you add that to another Host configuration like this:

    Host machine2
        HostName 192.168.0.42
        User username
        ssh -q -W %h:%p myserver

    This would mean that your ssh connection to 192.168.0.42 as user username would be made via myserver and all data would be forwarded via myserver. This can also be used to connect via a particular server all the time if it is added to the „Host *“ section of you ssh config.

    SSH config for hidden Tor service SSH server

    Something else I like to use ssh config for, is to connect to SSH via a tor Onion service, which I like to use for headless small computers (portable computer without a screen) like a Raspberry Pi that I connect to some network and want to SSH into them without knowing their IP address in the LAN. The setup of a Tor hidden service can be found online, the basic way it works is that your raspberry pi will tunnel to the Internet and be reachable via a hidden service. 

    Host raspberrypi
	HostName ap37csfkkmkksmkktllhjkcqd.onion 
        User Pi
        Port 22
        VerifyHostKeyDNS no
        ProxyCommand /bin/nc -xlocalhost:9050 -X5 %h %p

    HostName now contains the .Onion domain of the hidden service.

    I use VerifyHostKeyDNS to avoid warning about changed DNS for that host. I am not sure if it is actually needed, but at some point I added it.

    The ProxyCommand parameter now is the important thing. I requires you to have Tor running as a client, this should be possible by „sudo apt-get install tor && sudo systemctl enable tor.service –now“ Probably have to run it in two commands.
    This uses netcat (nc) to forward all packets from ssh to nc to the local port 9050 (standard Tor Socks port).
    I never noted what exactly the parameters for nc do. 🙂 if anyone know please let me know. I tried a quick search but it did not bring up any helpful results.

    Port must contain the port the hidden service ssh is available at, this can be different from the clients ssh daemon/server but is defined by the hidden service configuration in the torrc config file.

    Have a look in the manpage of ssh by typing „man ssh“ in your command line or open the manpage online. The ssh_config manpage is also very helpful.

    autossh

    I lately started to use autossh in combination with remote screen sessions, screen is the tool to use if you work remotely and do something that you want to continue after a possible connection interruption. There is tmux as well, but I prefer screen. Please have a look at both and choose which you prefer and use it.

    Quote from the manpage of autossh:

    autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic.

    It basically improves the interruption resistivity of ssh by wrapping some other things around the connection and by using some special parameters. As I said I work remotely often and also I use mobile connections often on trains as well, I uses mosh (MObile SHell) before, but it is not working with screen that well, also it had some other downsides and to I ended up with autossh.

    Let me know if there is any faults in this summary about my ssh use and setup.

    Edit:
    I figured something else out, I always wanted to keep my ssh config in sync between some computers. The problem was the synced folder is not ~/.ssh but ~/some/other/folder and in there I have a „ssh_config“ file. So I tried to symlink that to my ~/.ssh which did not work (ln -s ~/some/other/folder/ssh_config ~/.ssh/config did not work) because auf some permission issue I think. Host in the file were just not found.
    Today I figured out that I can include another file in my ~/.ssh/config by adding the following line in my ssh config file:

    Include ~/some/other/folder/ssh_config

    This now allows me to sync ~/some/other/folder/ssh_config between computers and include that file in my ssh config files on any computer I like to.

    Keine Kommentare zu SSH config, ssh via jumphost and autossh (ssh to Tor hidden service)

  • ,

    VM runterfahren

    Image der VM vergrössern:

    sudo qemu-img resize /var/lib/libvirt/images/vm_disk.img +15GB

    Je nach Setup ist sudo nicht nötig, durch den Befehl wird die Diskimagedatei um 15 GB vergrĂ¶ĂŸert.

    VM starten

    Weiter geht es in der VM auf der Konsole.

    Partitioen mit parted anzeigen lassen: 

    parted -l

    Ausgabe so oder Àhnlich:
    Model: Virtio Block Device (virtblk)
    Disk /dev/vda: 53,7GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Disk Flags:

    Number Start End Size Type File system Flags
    1 1049kB 53,7GB 53,7GB primary btrfs boot

    Ich habe, bevor ich angefangen habe, die VM heruntergefahren und eine Kopie des Diskimage erstellt. Deswegen habe ich mich dann entschieden die Partition in der laufenden VM anzupassen, grundsÀtzlich sollte das bei Linux gehen. Auflösung, es hat geklappt. Also weiter.

    Disk auswÀhlen bei Start von Parted:

    sudo parted /dev/vda

    In parted the Partitionen nochmal anzeigen:

    print

    Ausgabe so oder Àhnlich:
    Model: Virtio Block Device (virtblk)
    Disk /dev/vda: 69,8GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Disk Flags:

    Number Start End Size Type File system Flags
    1 1049kB 53,7GB 53,7GB primary btrfs boot

    In parted die Partition (1) vergrĂ¶ĂŸern:

    resizepart 1

    Dann die Abfragen bestĂ€tigen, maximale GrĂ¶ĂŸe eingeben (max oder 100% scheinen nicht mehr zu funktionieren) und bestĂ€tigen das der Vorgang durchgefĂŒhrt werden soll trotz das die Partition gemountet ist.

    Zu guter Letzt noch Dateisystem (btrfs) vergrössern:

    sudo btrfs filesystem resize max /

    Anmerkung: Bei anderen Dateisystemen ist der Befehl ein anderer. Im Beispiel ist der Mountpoint „/“, ggf. muss die auch angepasst werden.

    Keine Kommentare zu Resize qemu VM disk

  • Did you know explainshell.com yet? Try it, for example with that command from last time

    sudo lsof -i -P -n | grep LISTEN

    this is an amazing web site I use often so see what commands do that I find somewhere. Try it.

    Keine Kommentare zu explainshell.com

  • ,

    It drives me nuts that I cannot copy something from a ssh terminal when I use vim. I have no clue what this visual mode is meant to do besides anoying my.

    My solution:

    echo "set mouse-=a" >> ~/.vimrc

    no more visual mode -> marking and copying text from vim by using the mouse works again.

    Keine Kommentare zu disable vim visual mode

  • A short one:

    Command

    lsof -i -P -n | grep LISTEN

    This will output all the ports the system is listing on. You might have to prepend „sudo“.

    So then the command will be:

    sudo lsof -i -P -n | grep LISTEN

    The list is long without the „grep“, it will filter the output and only show the lines which contain „LISTEN“.

    Edit:
    Oliver from Hubzilla (part of the Fediverse) pointed out that 

    netstat -anpl

    might be helpful too. The command shows all the listing sockets of the Linux machine it is run on. (I added the „l“ parameter to just show listing sockets.

    Keine Kommentare zu lsof – local open ports

  • This is kind of my cheat sheet, nothing fancy, just what I need.

    Command: 

    rsync -chavzP --stats 'host:~/RemotePath' ./LocalPath

    rsync – Befehl
    -c, –checksum
    -h, –human-readable
    -a, –archive
    -v, –verbose
    -z, –compress
    -P – equivalent to –partial –progress
    –stats – This tells rsync to print a verbose set of statistics on the file transfer
    ‚host:~/RemotePath‘ – is the source path (in this case a remote path.
    ./LocalPath – ist the destination

    Some more variants:

    from local to remote

    rsync -chavzP --stats ~/FileName.file 'host:~/path/'

    Leave the z if compression does not matter (in most cases it does not matter).

    rsync -chavP --stats ~/FileName.file 'host:~/path/'

    Rsync to MOVE a folder from A to B. Source will be deleted!!

    rsync -hcavP --stats --remove-source-files ./source ./destination_folder

    Rsync can be used from local to local to copy files. It can be used from ‚remote‘ to local or vom local to ‚remote‘, unfortunately it can not copy from remote to remote.

    Don’t forget to include the ticks around the remote (host) part.

    I this „rsync -chavzP –stats ~/FileName.file ‚host:~/path/’“ the „host“ is defined in my ~/.ssh/config, but you can also do something like this:
    „rsync -chavzP –stats ~/FileName.file ‚user@example.org:~/path/’“

    This even work via ssh jump hosts. I will write something about that soon.

    Keine Kommentare zu rsync