For fun, convenience and a chance to learn shell scripting for myself, I’ve been working on a collection of scripts to semi-automate the backup of my computer, two network connected Raspberry Pi’s and my Android phone with Termux.

The main script basically runs a bunch of remote backup scripts, then copies those remote backups to a dedicated partition on my computer and finally creates a copy of that partition to an external drive connected to my computer. I use rsync and it’s dry run feature so I am able to examine any file changes which has been super useful to me for catching mistakes and issues as I’ve been learning how to self-host over the past half year.

I have a simplified version of those backup scripts that makes a copy of my /home directory:

#!/bin/sh



# VARIABLES
## ENABLE FILE TRANSFER: Change variable `DRY_RUN` to `#DRY_RUN` to enable file tranfers
DRY_RUN="--dry-run" # Disables all file transfers

## PATHS (USER DEFINED)
SOURCE_DIRECTORY_PATH="/home"
SOURCE_DIRECTORY_PATH_EXCLUSIONS="--exclude=lost+found --exclude=.cache/*"
BACKUP_NAME="home"
BACKUP_BASE_PATH="/backup"

## PATHS (SCRIPT DEFINED/DO NOT TOUCH)
SOURCE_DIR="${SOURCE_DIRECTORY_PATH}/"
DESTINATION_DIR="${BACKUP_BASE_PATH}/${BACKUP_NAME}/"

## EXCLUSIONS (SCRIPT DEFINED/DO NOT TOUCH)
EXCLUDE_DIR="${SOURCE_DIRECTORY_PATH_EXCLUSIONS}"

## OPTIONS (SCRIPT DEFINED/DO NOT TOUCH)
OPTIONS="--archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress"
OPTIONS_EXTRA="--delete --numeric-ids"



# FUNCTIONS
## SPACER
SPACER() {
    printf "\n\n\n\n\n"
}

## RSYNC ERROR WARNINGS
ERROR_WARNINGS() {
    if [ "$RSYNC_STATUS" -eq 0 ]; then

        # SUCCESSFUL
        printf "\nSync successful"
        printf "\nExit status(0): %s\n" "$RSYNC_STATUS"

    else
        # ERRORS OCCURED
        printf "\nSome error occurred"
        printf "\nExit status(0): %s\n" "$RSYNC_STATUS"
    fi
}

## CONFIRMATION (YES/NO)
CONFIRM_YESNO() {
    while true; do
        prompt="${1}"
        printf "%s (Yes/No): " "${prompt}" >&2 # FUNCTION CALL REQUIRES TEXT PROMPT ARGUMENT
        read -r reply
        case $reply in
            [Yy]* ) return 0;; # YES
            [Nn]* ) return 1;; # NO
            * ) printf "Options: y / n\n";;
        esac
    done
}



##### START

# CHECK FOR ROOT
if ! [ "$(id -u)" = 0 ]; then

    # EXIT WITH NO ACTIONS TAKEN
    printf "\nRoot access required\n\n"
    return

else
    printf "\nStarting backup process..."

    # ${SOURCE_DIR} TO ${DESTINATION_DIR} DRY RUN
    SPACER
    printf "\nStarting %s dry run\n" "${SOURCE_DIR}"
    rsync --dry-run ${OPTIONS} ${OPTIONS_EXTRA} ${EXCLUDE_DIR} "${SOURCE_DIR}" "${DESTINATION_DIR}"
    RSYNC_STATUS=$?
    ERROR_WARNINGS

    # CONFIRM ${SOURCE_DIR} TO ${DESTINATION_DIR} BACKUP
    SPACER
    if CONFIRM_YESNO "Proceed with ${SOURCE_DIR} backup?"; then

        # CONTINUE ${SOURCE_DIR} TO ${DESTINATION_DIR} BACKUP & EXIT
        printf "\nContinuing %s backup\n" "${SOURCE_DIR}"
        rsync ${DRY_RUN} ${OPTIONS} ${OPTIONS_EXTRA} ${EXCLUDE_DIR} "${SOURCE_DIR}" "${DESTINATION_DIR}"
        RSYNC_STATUS=$?
        ERROR_WARNINGS

        printf "\n%s backup completed\n\n" "${SOURCE_DIR}"
        return

    else
        # SKIP ${SOURCE_DIR} TO ${DESTINATION_DIR} BACKUP & EXIT
        printf "\n%s backup skipped\n\n" "${SOURCE_DIR}"
        return
    fi
fi

##### FINISH

I would like to adapt this script so that I can add multiple copies of the following variables:

## PATHS (USER DEFINED)
SOURCE_DIRECTORY_PATH="/home"
SOURCE_DIRECTORY_PATH_EXCLUSIONS="--exclude=lost+found --exclude=.cache/*"
BACKUP_NAME="home"
BACKUP_BASE_PATH="/backup"

without having to make multiple copies of the following commands within the running script:

    # ${SOURCE_DIR} TO ${DESTINATION_DIR} DRY RUN
    SPACER
    printf "\nStarting %s dry run\n" "${SOURCE_DIR}"
    rsync --dry-run ${OPTIONS} ${OPTIONS_EXTRA} ${EXCLUDE_DIR} "${SOURCE_DIR}" "${DESTINATION_DIR}"
    RSYNC_STATUS=$?
    ERROR_WARNINGS

I’m mainly just looking for a way to avoid touching the script commands itself so I don’t have to change the variable names for each additional directory I want to add. I’m not sure what that would be called or where to look. Any help would be greatly appreciated.

  • mcmodknower@programming.devEnglish
    7·
    2 months ago

    You probably want something like the array variables from bash. You can find out more about them in the manpage, in the Arrays section.

    Since you are using sh, you don’t have direct access to them. If you want to use them, change the shebang to #!/usr/bin/env bash.

    • confusedpuppy@lemmy.dbzer0.comOPEnglish
      1·
      2 months ago

      I was hoping to stay using shell since I am using these scripts on both Alpine Linux and Debian. I have a preference for learning how to keep the script portable which is why I tried my best to follow POSIX standards. Shellcheck was quite useful for that when I rewrote all the scripts I wrote.

      If I must, it wouldn’t be hard add bash to Alpine although I prefer to keep my web facing server as minimal as possible.

      Although I can look into arrays, that seems like a good spot to start looking.

      • sunshine@lemmy.mlEnglish
        3·
        2 months ago

        you will still be invoking this from the shell using the solution that person described. the line of code that they wrote for you starting with #! is called a shebang, and it’s job is to tell the computer what interpreter to use to run your script if you execute the script, rather than executing bash and passing the script as an argument. that is, ./script.sh rather than bash my_script.sh. note, you will need to run chmod +x my_script.sh first in order to flag the script file as okay to execute.

        • confusedpuppy@lemmy.dbzer0.comOPEnglish
          2·
          2 months ago

          How does setting the shebang to bash work when Alpine linux doesn’t ship with bash? Currently the bash command just leads to a script with only a few lines of code in it that leads back to itself.

          I know I can simply download bash but I enjoy the concept of being able to talk to as many devices as possible with as few layers of abstraction as possible. It’s why I chose to do things the POSIX way, it’s interesting to me.

          • sunshine@lemmy.mlEnglish
            2·
            2 months ago

            I apologize, I misread “want to keep using shell” as “want to keep using the shell” and I figured that you probably meant you didn’t get what the shebang was for.

  • NullPointer@programming.devEnglish
    2·
    2 months ago

    try storing the paths to backup in a dot-file; one path per line. then, in the script, iterate that file line by line backing up the paths.

    • confusedpuppy@lemmy.dbzer0.comOPEnglish
      1·
      1 month ago

      I ended up using dot-files like you suggested. Instead of multiple source/destination locations within one dot-file, I’ve set it up to read multiple dot-files that contains one source/destination location each.

      Now I can use the command dir-sync /path/to/directory/.sync1 /path/to/directory/.sync2 or dir-sync /path/to/directory/* which is close enough to how I imagined it working.

      The challenge is that POSIX shell only has one array to work with. I may be able to inefficiently use more arrays, and I might try just for fun in the future but for now I have something that functions the way I want it to. I would have to use something like Bash if I intend to work more in depth with arrays though.

      .sync-0 example 
      SOURCE_DIRECTORY_PATH="/home"
SOURCE_DIRECTORY_PATH_EXCLUSIONS="--exclude=.sync* --exclude=lost+found --exclude=.cache/*"
DESTINATION_DIRECTORY_PATH="/media/archive/home"
      dir-sync script 
      #!/bin/sh



# VARIABLES
## ENABLE FILE TRANSFER: Change variable `DRY_RUN` to `#DRY_RUN` to enable file tranfers
DRY_RUN="--dry-run" # Disables all file transfers

## OPTIONS (SCRIPT DEFINED/DO NOT TOUCH)
OPTIONS="--archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress"
OPTIONS_EXTRA="--delete --numeric-ids"



# FUNCTIONS
## SPACER
SPACER() {
    printf "\n\n\n\n\n"
}

## RSYNC ERROR WARNINGS
ERROR_WARNINGS() {
    if [ "$RSYNC_STATUS" -eq 0 ]; then

        # SUCCESSFUL
        printf "\nSync successful"
        printf "\nExit status(0): %s\n" "$RSYNC_STATUS"

    else
        # ERRORS OCCURED
        printf "\nSome error occurred"
        printf "\nExit status(0): %s\n" "$RSYNC_STATUS"
    fi
}

## COPY INPUT STRING ARRAY
COPY_ARRAY()
{
    for i do
        printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/"
    done
    printf " "
}

## CONFIRMATION (YES/NO)
CONFIRM_YESNO() {
    while true; do
        prompt="${1}"
        printf "%s (Yes/No): " "${prompt}" >&2 # FUNCTION CALL REQUIRES TEXT PROMPT ARGUMENT
        read -r reply
        case $reply in
            [Yy]* ) return 0;; # YES
            [Nn]* ) return 1;; # NO
            * ) printf "Options: y / n\n";;
        esac
    done
}

## DRY RUN
DRY_RUN() {
    # ARRAY SETUP
    eval "set -- ${1}" # ${ARRAY}

    for i do
        # SET CURRENT VARIABLE FROM ARRAY
        current_array="${1}"

        # SET VARIABLES FROM SOURCE FILE
        . "${current_array}"
        source_dir="${SOURCE_DIRECTORY_PATH}"
        exclusions="${SOURCE_DIRECTORY_PATH_EXCLUSIONS}"
        destination_dir="${DESTINATION_DIRECTORY_PATH}"

        # RUN RSYNC DRY RUN COMMAND
        SPACER
        printf "\nStarting %s dry run\n" "${source_dir}"
        rsync --dry-run ${OPTIONS} ${OPTIONS_EXTRA}  ${exclusions} "${source_dir}"/ "${destination_dir}"/
        RSYNC_STATUS=$?
        ERROR_WARNINGS

        # SHIFT TO NEXT VARIABLE IN ARRAY
        shift 1
    done
}

## TRANSFER
TRANSFER() {
    # ARRAY SETUP
    eval "set -- ${1}" # ${ARRAY}

    for i do
        # SET CURRENT VARIABLE FROM ARRAY
        current_array="${1}"

        # SET VARIABLES FROM SOURCE FILE
        . "${current_array}"
        source_dir="${SOURCE_DIRECTORY_PATH}"
        exclusions="${SOURCE_DIRECTORY_PATH_EXCLUSIONS}"
        destination_dir="${DESTINATION_DIRECTORY_PATH}"

        # RUN RSYNC TRANSFER COMMAND
        SPACER
        printf "\nContinuing %s transfer\n" "${source_dir}"
        rsync ${DRY_RUN} ${OPTIONS} ${OPTIONS_EXTRA} ${exclusions} "${source_dir}"/ "${destination_dir}"/
        RSYNC_STATUS=$?
        ERROR_WARNINGS

        # SHIFT TO NEXT VARIABLE IN ARRAY
        shift 1
    done
}



##### START

# CHECK FOR ROOT
if ! [ "$(id -u)" = 0 ]; then

    # EXIT WITH NO ACTIONS TAKEN
    printf "\nRoot access required\n\n"
    return

else
    # ARRAY SETUP
    ARRAY=$(COPY_ARRAY "$@")

    printf "\nStarting transfer process..."

    # SOURCE TO DESTINATION DRY RUN
    DRY_RUN "${ARRAY}"

    # CONFIRM SOURCE TO DESTINATION TRANSFER
    SPACER
    if CONFIRM_YESNO "Proceed with transfer(s)?"; then

        # CONTINUE SOURCE TO DESTINATION TRANSFER & EXIT
        TRANSFER "${ARRAY}"

        printf "\nBackup(s) completed\n\n"
        return

    else
        # SKIP SOURCE TO DESTINATION TRANSFER & EXIT
        printf "\nBackup(s) skipped\n\n"
        return
    fi
fi

##### FINISH

      I now have a bunch of other ideas on how I want to improve the script now. Adding ssh/network options, a check to make sure there are no empty command arguments, an option to create a blank template .sync- file in a chosen directory and a reverse direction transfer option are a few things I can think of off the top off my head. Hopefully this array was the most difficult part to learn and understand.