Text

· · zu lesen 
cp -pRv --reflink=auto /source/folder /destination/folder

copy files and directories
-p same as –preserve=mode,ownership,timestamps
-R, -r, –recursive
copy directories recursively
-v verbose, show what is going on.

–reflink[=WHEN]
control clone/CoW copies.

This just copied 169GB of files in 20 seconds. This only works on supported filesystems, so for example by „#Btrfs, , 4.2, , , and “. See here.

Explainshell tells you this.

Text

· · zu lesen

I will not write a long explanation, just this short one.

The cronjob looks like this:

* * * * * sh -c 'flock -n /tmp/my.lockfile /home/user/scripts/ocr_pdf_scan.sh'

This will start sh (a shell, just like bash) and run the command after -c. The command flock will create a lock-file, to prevent that this cron job is started while there is still the previous cron jobs running.

The cron job runs every minute, so in crontab it has * * * * * (five stars in front of the command).

In the ocr_pdf_scan.sh I have this:

#!/bin/bash

find /home/user/scans/ocr -type f \( -iname "*.pdf" -and -not -iname "*_ocr.pdf" \) | while read file ; do OLDTimestamp=$(stat -c "%Y" "$file") && ocrmypdf -q -l deu+eng --rotate-pages --rotate-pages-threshold 8 -c -s "$file" /home/user/scans/ocr_ready/"$(basename "$file" ".pdf")_ocr.pdf" && touch -d @$OLDTimestamp /home/user/scans/ocr_ready/"$(basename "$file" ".pdf")_ocr.pdf" && rm "$file" ; done

exit 0

Edit 31.07.2023: Added the stat/touch part to reset the modification time after ocrmypdf processed the file.

This will search for all files ending in „*.pdf“ but not „*_ocr.pdf“, it will pipe these path and filenames through to the while read, which will execute for each found file the ocrmypdf for the path and file stored in $file and output to another folder, it will strip the .pdf from the filename and append „_ocr.pdf“ to the filename again.

I tried this with output to the same folder as well, it also works, but for some reason I set it up like this.

Text

· · zu lesen

Important

This fix can be applied before the reboot after the upgrade from Buster to Bullseye, if you missed it you need a MicroSD card reader and some Linux computer (another RaspberryPI would do) to edit the file.

In the file 

/etc/systemd/system/dhcpcd.service.d/wait.conf

there is a wrong path for the dhcpd, it has to be changed.

wrong (from buster):

/usr/lib/dhcpcd5/dhcpcd

correct and new on Bullseye:

/usr/sbin/dhcpcd

So you need to do this before the reboot, otherwise the RaspberryPi will not have a working WIFI or Ethernet connection after the reboot and will not allow you to login from remote by SSH.

If you still have the RaspberryPi running you can use the command from this Stackexchange comment as well.

Text

· · zu lesen

So to upgrade Raspbian, as it was previously called, from Buster to Bullseye went smooth an easy, I tried this with some Pi2B (armhf 32Bit) and with a Pi4. These two are headless systems, so I don’t know if this works well for any systems with window managers.

Better run these commands in screen, so when the SSH connection is interrupted you can reconnect and resume your work.

Edit: For the update there is a culprit, it will cause a problem with dhcpd, so before the reboot 13. (was point 12. before I added the remark below) please check this post.

Steps to update:

  1. sudo apt update
  2. sudo apt dist-upgrade -y
  3. sudo rpi-update
  4. sudo reboot
  5. sudo apt remove libc6-dev
  6. Edit the file /etc/apt/sources.list, change all occurrences of buster to bullseye.
  7. Edit the file /etc/apt/sources.list.d/raspi.list, change all occurrences of buster to bullseye.
  8. sudo apt update
  9. sudo apt install libgcc-8-dev gcc-8-base
    • Instead of sudo apt install better use my update script, just run ~/up „libgcc-8-dev gcc-8-base“ (incl. the „quotation marks“) and skip steps 8., 9., 10. and 11. It doesn’t matter when you already executed step 8..
  10. sudo apt dist-upgrade
  11. sudo apt autoclean
  12. Please edit the file /etc/systemd/system/dhcpcd.service.d/wait.conf, change the path in there from /usr/lib/dhcpcd5/dhcpcd to /usr/sbin/dhcpcd, see here for long version.
  13. sudo reboot

Somewhere along the way there will be a warning that „libc6-dev“ can not be upgrade (or so) as it breaks „libgcc-8-dev“, this is the reason for step 5. and 9..

Edit: On the Pi4 I got:

The following packages have unmet dependencies:
 libgcc1 : Depends: gcc-8-base (= 8.3.0-6+rpi1) but 8.4.0-7+rpi1 is to be installed

Which I solved by „sudo apt-get remove libgcc1“ and then I just restarted with either point 8. from above, but in my case with „~/up libgcc-8-dev“.

I prefer to update or install packages with this script.

first inspiration from here.

Text

· · zu lesen
  1. Add key
curl https://ftp-master.debian.org/keys/archive-key-10.asc | gpg --import 
gpg --export DC30D7C23CBBABEE | apt-key add -

The fingerprint (DC30D7C23CBBABEE) given is the one that the first command outputs, which is the fingerprint of the key downloaded by curl.

2. Add repo

echo "deb http://deb.debian.org/debian buster-backports main" |sudo tee /etc/apt/sources.list.d/deb_buster_backports.list
  1. run apt update/upgrade
  2. Install tor, I did this by:
apt-get install -o Dpkg::Options::="--force-confold" -y tor
Text

· · zu lesen

This is a solution for a , some without swap of some machine setup without swap where you need swap for something without wanting to fiddle with the drive partitions or so.

This is a quick and dirty „short manual“ for based environments (such as , or ).

First run this:

cat > dphys-swap.sh << EOF
#!/bin/bash
#~/up
apt-get install dphys-swapfile
echo "CONF_SWAPSIZE=2048"|tee -a /etc/dphys-swapfile
dphys-swapfile setup
dphys-swapfile swapon
systemctl restart dphys-swapfile.service
systemctl status dphys-swapfile.service
free -h
EOF

This will just create a short with the needed commands to run to get setup and running. Please alter the number 2048 to your need, it is the size of the swap memory that will be setup. The size of the swap should most likely by equal to the real memory installed, sometimes twice the size seems to be a good choice too.

If you don’t know your memory size you can find it out by running the command „free -h“, which will print the size of the physical memory installed and the below this the swap size(s).

After pasting the above code into your shell, which will create a file called „dphys-swap.sh“ in the current folder (you should be in your home folder!!), you need to make the shell script „dphys-swap.sh“ executable by running:

chmod +x dphys-swap.sh

Then you need to run the script like this:

sudo ~/dphys-swap.sh

It will ask for some confirmation(s) to install certain packages.

The last commands will output the status of the systemd service „dphys-swapfile.service“ and the last line will show you the memory and swap size again.

Remark:
In the script above I use „up“, my full update of the system. I just blogged that one too, have a look here.

Text

· · zu lesen

I have this small script deployed on any of my based systems, it just does all the update, upgrade, dist-upgrade, autoremove and autoclean in one run. 

#!/bin/sh 

if [ $# -eq 0 ]; then
  sudo sh -c "apt-get update && apt-get upgrade && apt-get dist-upgrade && apt-get update && apt-get autoremove && apt-get autoclean"
else
  sudo sh -c "apt-get update && apt-get upgrade && apt-get dist-upgrade && apt-get update && apt-get install $@ && apt-get autoremove && apt-get autoclean"
fi

I have this script deployed in a lazy manner, I just have it sitting in my users home folder and called it „up“, I did do a „chmod +x up“ and just start it like this:

~/up

It also accepts one several package names as an arguments and that be called like this:

sudo ~/up package1,package2

This will then install „package1“ and „package2“.

I am not sure where I found this, but I surely did find this somewhere in the internet and just took it from there, you thanks to who ever put this together.

Text

· · zu lesen

To do this you need two empty drives, with the guide below all data on the drive will be lost if it is not empty!

1. Install cryptsetup

apt-get install cryptsetup

2. Create the dm-crypt 

cryptsetup -y -v luksFormat /dev/sdb
cryptsetup -y -v luksFormat /dev/sdc

(Make sure to use the correct device. To show available devices use „lsblk“.)

3. Mount the cryptsetup disk

cryptsetup luksOpen /dev/sdb disk1
cryptsetup luksOpen /dev/sdc disk2

4. Check if this was successful

ls -l /dev/mapper/disk*

or 
sudo cryptsetup -v status disk1
sudo cryptsetup -v status disk2

5. Create BTRFS volume RAID 1, this means all data and meta-data is mirrored, which means if you use 2 drives with i.e. 4TB each you can only use 4TG over all in the BTRFS mounted volume.

mkfs.btrfs -m raid1 -d raid1 /dev/mapper/disk1 /dev/mapper/disk2

You could list more than two drive, for example to increase redundancy, but if you want to use more drive to increase storage you would have to use a different Raid level.

6. Once you created a multi-device file system, you can use any device in the FS for the mount command

mount -t btrfs -o noatime,compress=zstd /dev/mapper/disk1 /mnt/btrfs

7. Creat a subvolume for each purpose/for each data storage folder you want to use the RAID 1 for

btrfs subvolume create /mnt/btrfs/subvolume1_data

For example if you want to store you /srv/http and your /var/lib/mysql on the BTRFS RAID drive you should create one subvolume for each, of them, i.e.

btrfs subvolume create /mnt/btrfs/subvolume_http
btrfs subvolume create /mnt/btrfs/subvolume_mysql

Those subvolumes can than be mounted in the respective folders. Before you mount them you should set the permissions/ownership of the folder as same as their mount folder (in this case check permissions of /srv/www and /var/lib/mysql), then move all files and folders from the current folder to the subvolume (see how to mount it below) and then mount the subvolume to the folder).

8. Mount the subvolume

mount -t btrfs -o subvolume=subvolume1_data,noatime,compress=zstd /dev/mapper/disk1 /mnt/data

This uses zstd compression for the mount, this is invoked only by setting the mount parameter. For details on compression see here.

You can mount this subvolume, also the main btrfs volume, to any folder you want, lets assume you want to mount it to /home/$USER/data, so instead of the last command you want to do this (if already mounted before, just do „umount /mnt/data“ if you already mounted it)

mount -t btrfs -o subvolume=subvolume1_data,noatime,compress=zstd /dev/mapper/disk1 /home/$USER/data

(folder must exists, „mkdir /home/$USER/data“ if you want to create it)

Now the subvolume is available in /home/$USER/data, see „df -h“ to check.

The output will contain a line like this:
/dev/mapper/disk1 3.7T 17M 3.7T 1% /home/YOUR_USER_NAME/data

9. Change permission of the subvolume

chown -R $USER:$USER ~/data

Now you can write and read from the folder, you have full access to it, which also means you have full access to the subvolume content and the permissions are inherited by folders and files you create in the folder „data“.

10. Show BTRFS filesystem

sudo btrfs filesystem show

The output will look like this

Label: none  uuid: 690fbca6-f764-4873-9f2c-f60c40197a14
        Total devices 2 FS bytes used 656.00KiB
        devid    1 size 3.64TiB used 2.03GiB path /dev/mapper/disk2
        devid    2 size 3.64TiB used 2.03GiB path /dev/mapper/disk1

There is a great cheat sheet for BTRFS here.

Please leave a comment if you have a suggestion.