For fun, convenience and a chance to learn shell scripting for myself, I’ve been working on a collection of scripts to semi-automate the backup of my computer, two network connected Raspberry Pi’s and my Android phone with Termux.
The main script basically runs a bunch of remote backup scripts, then copies those remote backups to a dedicated partition on my computer and finally creates a copy of that partition to an external drive connected to my computer. I use rsync
and it’s dry run feature so I am able to examine any file changes which has been super useful to me for catching mistakes and issues as I’ve been learning how to self-host over the past half year.
I have a simplified version of those backup scripts that makes a copy of my /home directory:
#!/bin/sh
# VARIABLES
## ENABLE FILE TRANSFER: Change variable `DRY_RUN` to `#DRY_RUN` to enable file tranfers
DRY_RUN="--dry-run" # Disables all file transfers
## PATHS (USER DEFINED)
SOURCE_DIRECTORY_PATH="/home"
SOURCE_DIRECTORY_PATH_EXCLUSIONS="--exclude=lost+found --exclude=.cache/*"
BACKUP_NAME="home"
BACKUP_BASE_PATH="/backup"
## PATHS (SCRIPT DEFINED/DO NOT TOUCH)
SOURCE_DIR="${SOURCE_DIRECTORY_PATH}/"
DESTINATION_DIR="${BACKUP_BASE_PATH}/${BACKUP_NAME}/"
## EXCLUSIONS (SCRIPT DEFINED/DO NOT TOUCH)
EXCLUDE_DIR="${SOURCE_DIRECTORY_PATH_EXCLUSIONS}"
## OPTIONS (SCRIPT DEFINED/DO NOT TOUCH)
OPTIONS="--archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress"
OPTIONS_EXTRA="--delete --numeric-ids"
# FUNCTIONS
## SPACER
SPACER() {
printf "\n\n\n\n\n"
}
## RSYNC ERROR WARNINGS
ERROR_WARNINGS() {
if [ "$RSYNC_STATUS" -eq 0 ]; then
# SUCCESSFUL
printf "\nSync successful"
printf "\nExit status(0): %s\n" "$RSYNC_STATUS"
else
# ERRORS OCCURED
printf "\nSome error occurred"
printf "\nExit status(0): %s\n" "$RSYNC_STATUS"
fi
}
## CONFIRMATION (YES/NO)
CONFIRM_YESNO() {
while true; do
prompt="${1}"
printf "%s (Yes/No): " "${prompt}" >&2 # FUNCTION CALL REQUIRES TEXT PROMPT ARGUMENT
read -r reply
case $reply in
[Yy]* ) return 0;; # YES
[Nn]* ) return 1;; # NO
* ) printf "Options: y / n\n";;
esac
done
}
##### START
# CHECK FOR ROOT
if ! [ "$(id -u)" = 0 ]; then
# EXIT WITH NO ACTIONS TAKEN
printf "\nRoot access required\n\n"
return
else
printf "\nStarting backup process..."
# ${SOURCE_DIR} TO ${DESTINATION_DIR} DRY RUN
SPACER
printf "\nStarting %s dry run\n" "${SOURCE_DIR}"
rsync --dry-run ${OPTIONS} ${OPTIONS_EXTRA} ${EXCLUDE_DIR} "${SOURCE_DIR}" "${DESTINATION_DIR}"
RSYNC_STATUS=$?
ERROR_WARNINGS
# CONFIRM ${SOURCE_DIR} TO ${DESTINATION_DIR} BACKUP
SPACER
if CONFIRM_YESNO "Proceed with ${SOURCE_DIR} backup?"; then
# CONTINUE ${SOURCE_DIR} TO ${DESTINATION_DIR} BACKUP & EXIT
printf "\nContinuing %s backup\n" "${SOURCE_DIR}"
rsync ${DRY_RUN} ${OPTIONS} ${OPTIONS_EXTRA} ${EXCLUDE_DIR} "${SOURCE_DIR}" "${DESTINATION_DIR}"
RSYNC_STATUS=$?
ERROR_WARNINGS
printf "\n%s backup completed\n\n" "${SOURCE_DIR}"
return
else
# SKIP ${SOURCE_DIR} TO ${DESTINATION_DIR} BACKUP & EXIT
printf "\n%s backup skipped\n\n" "${SOURCE_DIR}"
return
fi
fi
##### FINISH
I would like to adapt this script so that I can add multiple copies of the following variables:
## PATHS (USER DEFINED)
SOURCE_DIRECTORY_PATH="/home"
SOURCE_DIRECTORY_PATH_EXCLUSIONS="--exclude=lost+found --exclude=.cache/*"
BACKUP_NAME="home"
BACKUP_BASE_PATH="/backup"
without having to make multiple copies of the following commands within the running script:
# ${SOURCE_DIR} TO ${DESTINATION_DIR} DRY RUN
SPACER
printf "\nStarting %s dry run\n" "${SOURCE_DIR}"
rsync --dry-run ${OPTIONS} ${OPTIONS_EXTRA} ${EXCLUDE_DIR} "${SOURCE_DIR}" "${DESTINATION_DIR}"
RSYNC_STATUS=$?
ERROR_WARNINGS
I’m mainly just looking for a way to avoid touching the script commands itself so I don’t have to change the variable names for each additional directory I want to add. I’m not sure what that would be called or where to look. Any help would be greatly appreciated.
You probably want something like the array variables from bash. You can find out more about them in the manpage, in the
Arrays
section.Since you are using sh, you don’t have direct access to them. If you want to use them, change the shebang to
.
I was hoping to stay using shell since I am using these scripts on both Alpine Linux and Debian. I have a preference for learning how to keep the script portable which is why I tried my best to follow POSIX standards.
Shellcheck
was quite useful for that when I rewrote all the scripts I wrote.If I must, it wouldn’t be hard add bash to Alpine although I prefer to keep my web facing server as minimal as possible.
Although I can look into arrays, that seems like a good spot to start looking.
you will still be invoking this from the shell using the solution that person described. the line of code that they wrote for you starting with
#!
is called a shebang, and it’s job is to tell the computer what interpreter to use to run your script if you execute the script, rather than executing bash and passing the script as an argument. that is,./script.sh
rather thanbash my_script.sh
. note, you will need to runchmod +x my_script.sh
first in order to flag the script file as okay to execute.How does setting the shebang to bash work when Alpine linux doesn’t ship with bash? Currently the bash command just leads to a script with only a few lines of code in it that leads back to itself.
I know I can simply download bash but I enjoy the concept of being able to talk to as many devices as possible with as few layers of abstraction as possible. It’s why I chose to do things the POSIX way, it’s interesting to me.
I apologize, I misread “want to keep using shell” as “want to keep using the shell” and I figured that you probably meant you didn’t get what the shebang was for.
try storing the paths to backup in a dot-file; one path per line. then, in the script, iterate that file line by line backing up the paths.
I ended up using dot-files like you suggested. Instead of multiple source/destination locations within one dot-file, I’ve set it up to read multiple dot-files that contains one source/destination location each.
Now I can use the command
dir-sync /path/to/directory/.sync1 /path/to/directory/.sync2
ordir-sync /path/to/directory/*
which is close enough to how I imagined it working.The challenge is that POSIX shell only has one array to work with. I may be able to inefficiently use more arrays, and I might try just for fun in the future but for now I have something that functions the way I want it to. I would have to use something like Bash if I intend to work more in depth with arrays though.
.sync-0 example
SOURCE_DIRECTORY_PATH="/home" SOURCE_DIRECTORY_PATH_EXCLUSIONS="--exclude=.sync* --exclude=lost+found --exclude=.cache/*" DESTINATION_DIRECTORY_PATH="/media/archive/home"
dir-sync script
#!/bin/sh # VARIABLES ## ENABLE FILE TRANSFER: Change variable `DRY_RUN` to `#DRY_RUN` to enable file tranfers DRY_RUN="--dry-run" # Disables all file transfers ## OPTIONS (SCRIPT DEFINED/DO NOT TOUCH) OPTIONS="--archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress" OPTIONS_EXTRA="--delete --numeric-ids" # FUNCTIONS ## SPACER SPACER() { printf "\n\n\n\n\n" } ## RSYNC ERROR WARNINGS ERROR_WARNINGS() { if [ "$RSYNC_STATUS" -eq 0 ]; then # SUCCESSFUL printf "\nSync successful" printf "\nExit status(0): %s\n" "$RSYNC_STATUS" else # ERRORS OCCURED printf "\nSome error occurred" printf "\nExit status(0): %s\n" "$RSYNC_STATUS" fi } ## COPY INPUT STRING ARRAY COPY_ARRAY() { for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" done printf " " } ## CONFIRMATION (YES/NO) CONFIRM_YESNO() { while true; do prompt="${1}" printf "%s (Yes/No): " "${prompt}" >&2 # FUNCTION CALL REQUIRES TEXT PROMPT ARGUMENT read -r reply case $reply in [Yy]* ) return 0;; # YES [Nn]* ) return 1;; # NO * ) printf "Options: y / n\n";; esac done } ## DRY RUN DRY_RUN() { # ARRAY SETUP eval "set -- ${1}" # ${ARRAY} for i do # SET CURRENT VARIABLE FROM ARRAY current_array="${1}" # SET VARIABLES FROM SOURCE FILE . "${current_array}" source_dir="${SOURCE_DIRECTORY_PATH}" exclusions="${SOURCE_DIRECTORY_PATH_EXCLUSIONS}" destination_dir="${DESTINATION_DIRECTORY_PATH}" # RUN RSYNC DRY RUN COMMAND SPACER printf "\nStarting %s dry run\n" "${source_dir}" rsync --dry-run ${OPTIONS} ${OPTIONS_EXTRA} ${exclusions} "${source_dir}"/ "${destination_dir}"/ RSYNC_STATUS=$? ERROR_WARNINGS # SHIFT TO NEXT VARIABLE IN ARRAY shift 1 done } ## TRANSFER TRANSFER() { # ARRAY SETUP eval "set -- ${1}" # ${ARRAY} for i do # SET CURRENT VARIABLE FROM ARRAY current_array="${1}" # SET VARIABLES FROM SOURCE FILE . "${current_array}" source_dir="${SOURCE_DIRECTORY_PATH}" exclusions="${SOURCE_DIRECTORY_PATH_EXCLUSIONS}" destination_dir="${DESTINATION_DIRECTORY_PATH}" # RUN RSYNC TRANSFER COMMAND SPACER printf "\nContinuing %s transfer\n" "${source_dir}" rsync ${DRY_RUN} ${OPTIONS} ${OPTIONS_EXTRA} ${exclusions} "${source_dir}"/ "${destination_dir}"/ RSYNC_STATUS=$? ERROR_WARNINGS # SHIFT TO NEXT VARIABLE IN ARRAY shift 1 done } ##### START # CHECK FOR ROOT if ! [ "$(id -u)" = 0 ]; then # EXIT WITH NO ACTIONS TAKEN printf "\nRoot access required\n\n" return else # ARRAY SETUP ARRAY=$(COPY_ARRAY "$@") printf "\nStarting transfer process..." # SOURCE TO DESTINATION DRY RUN DRY_RUN "${ARRAY}" # CONFIRM SOURCE TO DESTINATION TRANSFER SPACER if CONFIRM_YESNO "Proceed with transfer(s)?"; then # CONTINUE SOURCE TO DESTINATION TRANSFER & EXIT TRANSFER "${ARRAY}" printf "\nBackup(s) completed\n\n" return else # SKIP SOURCE TO DESTINATION TRANSFER & EXIT printf "\nBackup(s) skipped\n\n" return fi fi ##### FINISH
I now have a bunch of other ideas on how I want to improve the script now. Adding ssh/network options, a check to make sure there are no empty command arguments, an option to create a blank template .sync- file in a chosen directory and a reverse direction transfer option are a few things I can think of off the top off my head. Hopefully this array was the most difficult part to learn and understand.