Automatic backup of config using a "pull method"

I will improve it in the near future after I have it restructured. It's then also easier to implement little changes
 
I will improve it in the near future after I have it restructured. It's then also easier to implement little changes

I have restructured the script and tested it against the cluster of modems I am supervising.

It relies on 'httping' to be installed


The script can take command line parameters now...
  • -c use current directory instead of default
  • -d <folder> use <folder> instead of default
  • comma delimited parameters for 1 modem (same syntax as the list)

If no parameters for a modem are given, it will search for a list containing modem parameters in the designated folder and parse that list and fetch both the Freetz config and the AVM-config.
The parameters for 1 modem can be as basic as its external address. It will then use the default values for the rest....

The AVM-config is password-protected with the external password.
The files will be placed in a folder and get a little check. If the config is (about) the same as the last one fetched, it will get discarded.
This way you can just create a symlink to the script in /etc/cron.daily and you will always have a folder with different configs.
You can easily see when a config has been changed.

I'm placing those files in a password protected website where I can immediately download one of the configs I need....
Maybe it will save my ass one day...

It can now also get the AVM-config if there's no freetz on it.
If it doesn't detect the Freetz interface it will do an AVM only.
If you don't want to even try the Freetz-port you can define it as port 0


The script could do with some better logging and maybe there are some bugs there.

If you use the script I would appreciate some feedback.
I also would like to know if I should bother posting updates here.
Thanks Daniel and MaxMuster for the URL's I needed to get..


Update May 18 17:30
Added better handling for "#"
You can put comments at the end of the modemline and of course in the beginning
Everything behind the # will be ignored...

Also added some code to prevent duplicate AVM-configs (in case Freetz is not used)

Update May 18 00:00
Not only remove the path, but also the extension from the name of the script to use as the name for modemlist

Update May 19 12:00
I should spend my holidays better, really...
Another improvement to make the detection for changes a bit better and more sensible.
If only an AVM-config is made, the script kept the config too often. I now check for big differences by ignoring the binary data in the config.
If, however, the latest config is older than 30 days the script does an exact comparison.....

Update May 20 13:15
The script still made some unnecessary backups, so I investigated it and improved the comparison. Added some extra logging...
If used as a cronjob, it will only output data if a changes is found....

Some examples:

Simple help
Code:
freetz_getconfigs -?
Get config in default folder using a list called freetz_getconfigs.list containing all the modems
Code:
freetz_getconfigs
Get config in folder /opt from host remote.domain.com using all default values
Code:
freetz_getconfigs -d /opt remote.domain.com
Get config in current folder from host remote.domain.com using all default values
Code:
freetz_getconfigs -c  remote.domain.com
Get config in current folder from 192.168.178.1 using port 81 for Freetz interface and port 451 for the AVM interface
Code:
freetz_getconfigs -c  mymodem,192.168.178.1,,81,,,80
Get config in current folder from 192.168.178.1 from LAN using a plain Jane AVM Fritzbox without Freetz
Code:
freetz_getconfigs -c  mymodem,192.168.178.1,,0,,,80
Get config in current folder from host remote.domain.com using port 451 for the AVM interface
Code:
freetz_getconfigs -c  remote.domain.com,,,,,,451


# cat /usr/local/sbin/freetz_getconfigs
Code:
#!/bin/sh

# Define constants
DATESTAMP=`date +%Y-%m-%d.%H-%M`
FOLDER=/var/www/vhosts/domain.com/freetz

USER_DEFAULT=admin
PASS_DEFAULT=secret
PORT_DEFAULT=6080
PORTAVM_DEFAULT=450
MINSIZE=20000
HEADLESS=
tty >/dev/null || HEADLESS=true

while getopts cd: opt
do
  case $opt in
    c)   FOLDER="`pwd`" ;;
    d)   FOLDER="$OPTARG";;
    ?)   printf "Usage: %s [-c] [-d <folder>] [<HOST>],[<IP>],[PASS_FREETZ],[<PORT_FREETZ>],[<USER>],[PASS_EXTERNAL],[PORT_AVM]\n" $0
    exit 2;;
  esac
done
shift `expr ${OPTIND} - 1`


# Subroutines

age_of_file ()
{
  if [ ! -e "$1" ] ; then
    echo 0
  else
    AGE_IN_SECONDS=$((`date +%s` - `date +%s -r "$1"` ))
    echo $((${AGE_IN_SECONDS} / 86400))
  fi
}

httpspeed ()
{
  # Returns no value and  errorlevel 1 if site is unreachable or too slow
  # Returns the speed in milliseconds if it's below 2 seconds
  _SPEED=`httping -c1 -t1 ${1} 2>/dev/null | \
         egrep '.+/.+/.+/.+/.+' | tail -n1 | \
         awk -F/ '{print $4}' | awk -F. '{print $1}'`

  if [ -z "${_SPEED}" ] ; then
    echo "${URL} is unreachable" >&2
  elif [ ${_SPEED} -gt 2000 ] ; then
    echo "${URL} is slow (${_SPEED} ms)" >&2
  else
    echo ${_SPEED}
    return 0
  fi
  return 1
}


get_avm_config ()
{
  [ ${PORTAVM} -eq 0 ] && return

  TMPDIR2=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
  if ! cd ${TMPDIR2} 2>/dev/null ; then
    echo "Error changing current directory to ${TMPDIR2}. You should never see this message!" >&2
  else

    # It's intended to be called from WAN (https)
    # If PORTAVM begins with 8 it assumes LAN and turns to http
    PROTO=https
    echo ${PORTAVM} | grep -q '^8' && PROTO=http
    URL="${PROTO}://${IP}:${PORTAVM}"

    if httpspeed ${URL} >/dev/null ; then
      # get challenge key from FB

      _CHALLENGE=`curl -s -k \
                       --user ${USER}:${PASSE} \
                       "${URL}/login.lua" | \
                       grep 'challenge' | egrep '[0-9a-f]{8}' | \
                       awk -F= '{print $2}' | tail -n1 | egrep -o '[0-9a-f]{8}'`

      if [ -z "${_CHALLENGE}" ] ; then
        echo "Error getting challenge for ${HOST} using ${URL}" >&2
      else
        # echo "Received Challenge \"${_CHALLENGE}\"" >&2
        # build md5 from challenge key and password
        _MD5=`echo -n ${_CHALLENGE}"-"${PASSE} | \
              iconv -f ISO8859-1 -t UTF-16LE   | \
              md5sum -b | awk '{print substr($0,1,32)}'`

        # assemble challenge key and md5
        _RESPONSE=${_CHALLENGE}"-"${_MD5}

        # get sid for later use
        _SID=`curl -i -s -k \
                    --user ${USER}:${PASSE} \
                    -d 'response='${_RESPONSE} \
                    -d 'page=' \
                    -d 'username='${USER} \
                    ${URL}/login.lua | \
                    grep "Location:" | awk -F'=' {' print $NF '}`

        if ! curl -s -k \
                  --user ${USER}:${PASSE} \
                  --form 'sid='${_SID} \
                  --form 'ImportExportPassword='${PASSE} \
                  --form 'ConfigExport=' \
                  ${URL}/cgi-bin/firmwarecfg >TMP1 ; then

          echo "Error getting .export for ${HOST} (no data)" >&2
        else
          if [ `stat -c%s TMP1` -lt ${MINSIZE} ] ; then
            echo "Error getting .export for ${HOST} using ${URL} (too small)" >&2
            echo "SID: \"${_SID}\" CHALLENGE: \"${_CHALLENGE}\"  MD5: \"${_MD5}\"  RESPONSE: \"${_RESPONSE}\"" >&2
          else
            chown ${FUSER}:${FGROUP} TMP1
            mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
            LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
                          grep "${IP}.*\.export" | xargs -I{} stat -c '%Y %n' {} | \
                          sort -rn | head -n1 | awk '{print $2}'`

            if [ -z "${LATESTCONFIG}" ] ; then                     # No previous .export is found
              echo "Saving \"${DEST_BASE}.export\" because no previous AVM-config has been found" >&2
              mv TMP1 "${DEST_BASE}.export"
            else
              if [ ${TRIED_FREETZ} ] ; then      # The Freetz comparison is better, so make that leading
                if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
                  echo "Saving \"${DEST_BASE}.export\" because Freetz config has changed" >&2
                  mv TMP1 "${DEST_BASE}.export"
                fi
              else
                if [ `age_of_file "${LATESTCONFIG}"` -gt 30 ] ; then      # Compare EXACTLY if age is older than 30 days
                  if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
                    echo "Saving \"${DEST_BASE}.export\" because latest AVM-config was older than 30 days and is different" >&2
                    mv TMP1 "${DEST_BASE}.export"
                  fi
                else
                  # remove BINARY data and timestamps before making a diff
                  sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d' "${LATESTCONFIG}" >TMP2
                  sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d' TMP1              >TMP3
                  if ! diff TMP2 TMP3 >/dev/null 2>&1 ; then                  # Compare the 2 stripped configs
                    echo "Saving \"${DEST_BASE}.export\" because AVM-config has changed" >&2
                    diff TMP2 TMP3
                    mv TMP1 "${DEST_BASE}.export"
                  fi
                fi
              fi
            fi
          fi
        fi
      fi
    fi
  fi
  cd "${FOLDER}"
  rm -rf ${TMPDIR2}
}

patch_files ()
{
  # remove or patch some of the untarred files as they spoil the check for a changed config
  if cd "$1" >/dev/null 2>&1 ; then
    rm -f stat.cfg chrony.drift multid.leases 2>/dev/null
    rm -f fx_cg fonctrl phonebook voipd_call_stat 2>/dev/null

    sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' voip.cfg  # remove datestamp in voip.cfg
    sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' ar7.cfg   # remove datestamp in ar7.cfg
    sed -i 's/ *\/\*AVM\*\///g' ar7.cfg                     # remove the /*AVM*/ tag in ar7.cfg
    sed -i 's/; *$/;/g' ar7.cfg                             # remove trailing spaces after a semicolon
    cd - >/dev/null 2>&1
  fi
}

get_freetz_config ()
{
  # Fetch config-file with wget in background
  # Somehow it doesn't listen to the 6 seconds timeout I'm giving it
  wget -q --timeout=6                              \
       --http-user=${USER} --http-password=${PASS} \
       ${URL}/cgi-bin/backup/do_backup.cgi         \
       -O TMP_CONFIG 2>/dev/null &
  # Little wait loop to give wget the time to fetch the file
  n=1
  while sleep 1 ; do
    [ `stat -c%s TMP_CONFIG` -ge ${MINSIZE} ] && break
    [ $n -gt 7 ]                              && break
    let n+=1
  done

  if [ `stat -c%s TMP_CONFIG` -gt 300 ] ; then
    # If I received at least some data, then wait some more and it might get everything
    sleep 2
  else
    echo "wget didn't get any data from host \"${HOST}\"" >&2
    exec 2>/dev/null
    kill -9 %1 2>/dev/null
    exec 2>&2
    return 1
  fi
  return 0
}

keep_config_if_different ()
{
  LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
                      grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | \
                      sort -rn | head -n1 | awk '{print $2}'`

  # If no older config is found
  if [ -z "${LATESTCONFIG}" ] ; then
    echo "Saving \"${DEST_BASE}.tgz\" because no previous Freetz config has been found" >&2
    mv TMP_CONFIG "${DEST_BASE}.tgz"
    get_avm_config
  else

    TRIED_FREETZ=1  # A flag for get_avm_config to know if the freetz config has been tried or not
    mkdir f1   # folder to untar the current config
    mkdir f2   # folder to untar the previous config
    if ! tar xzf TMP_CONFIG -C f1 2>/dev/null ; then
      echo "I just downloaded an invalid gzipped tar-file" >&2
      rm -f TMP_CONFIG
    else
      if ! tar xzf ${LATESTCONFIG} -C f2 ; then
        echo "A previously downloaded config turns out to be invalid... very strange... (${LATESTCONFIG})" >&2
      else
        # Remove files that are likely to be changed, but don't contain important config
        patch_files f1/var_flash
        patch_files f2/var_flash

        if ! diff f1/var_flash f2/var_flash >/dev/null 2>&1 ; then
          echo "Saving \"${DEST_BASE}.tgz\" because Freetz config changed" >&2
          mv TMP_CONFIG "${DEST_BASE}.tgz"
          # Now get the AVM config too
          get_avm_config
        elif [ `find "${DEST_FOLDER}" -maxdepth 1 -type f -name \*.export | wc -l` -lt 1 ] ; then
          # No previous AVM config exists, so try and fetch one anyhow
          get_avm_config
       fi
     fi
   fi
 fi
}

get_both_configs ()
{
  PROTO=http
  echo ${PORT} | grep -q '^4[456][0-9]' && PROTO=https
  URL="${PROTO}://${IP}:${PORT}"

  if ! httpspeed ${URL} >/dev/null ; then
    # Freetz is not reacting, maybe it is a plain jane AVM Fritzbox
    # I will not be able to see if a config has changed.......
    echo "No reaction from Freetz.... will try AVM now.."
    get_avm_config
  else
    TMPDIR1=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`

    if ! cd ${TMPDIR1} ; then
      echo "Error changing folder to ${TMPDIR1}" >&2
    else
      touch TMP_CONFIG

      get_freetz_config

      if [ `stat -c%s TMP_CONFIG` -lt ${MINSIZE} ] ; then
        echo "Error getting config from ${HOST} on ${IP}" >&2
      else

        # I think we're getting successful here, let's create a folder already
        chown ${FUSER}:${FGROUP} TMP_CONFIG
        mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"

        keep_config_if_different
      fi
    fi
    cd "${FOLDER}"
    rm -rf ${TMPDIR1}
  fi
}


# Main Program starts here

if ! cd "${FOLDER}" ; then
  echo "${FOLDER} does not exist" >&2
  exit 1
fi

BASENAME="${0//*\/}"                           # Strip Path of script name
BASENAME="${BASENAME%.*}"                      # Strip extension
CONFIGSOURCE="${FOLDER}/${BASENAME}.list"      # Construct modemlist from script name
CONFIGFILE=`mktemp`                            # temp file for sanitized modemlist

if [ -n "$*" ] ; then
  echo "$*" >"${CONFIGFILE}"                     # push the commandline parameters in the configfile
elif [ -e "${CONFIGSOURCE}" ] ; then
  awk -F# '{print $1}' "${CONFIGSOURCE}" | \
      sed '/^ *$/d' >"${CONFIGFILE}"             # Get all the data before # and remove empty lines
else
  echo "Configfile: \"${CONFIGSOURCE}\" does not exist" >&2
  exit 1
fi

if [ ! -s "${CONFIGFILE}" ] ; then
  echo "No workable data in \"${CONFIGSOURCE}\"" >&2
  exit 1
fi

# Main Loop
FGROUP=`stat -c%G .`
FUSER=`stat -c%U .`

while read MODEMLINE ; do
  HOST="`echo "${MODEMLINE}"    | awk -F, '{print $1}'`"
  IP="`echo "${MODEMLINE}"      | awk -F, '{print $2}'`"
  PASS="`echo "${MODEMLINE}"    | awk -F, '{print $3}'`"
  PORT="`echo "${MODEMLINE}"    | awk -F, '{print $4}' | tr -cd '0-9'`"
  USER="`echo "${MODEMLINE}"    | awk -F, '{print $5}'`"
  PASSE="`echo "${MODEMLINE}"   | awk -F, '{print $6}'`"
  PORTAVM="`echo "${MODEMLINE}" | awk -F, '{print $7}' | tr -cd '0-9'`"

  HOSTORG="${HOST}"
  TRIED_FREETZ=

  if [ -z "${IP}" ] ; then
    HOST=`echo "${HOST}" | tr 'A-Z' 'a-z' | \
          egrep -o '[a-z0-9.-]+\.[a-z0-9]+'`     # Loosely Sanitize domain if no IP is given
  fi
  if [ -z "${HOST}" ] ; then
    echo "${HOSTORG} is not a valid host" >&2
  else
    [ ${HEADLESS} ] || echo ${HOST}
    if [ -z "${IP}" ] ; then
      # If HOST is so clearly an IP-address then use that instead....
      if echo "${HOST}" | egrep -q '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$' ; then
        IP=${HOST}
      else
        IP=`host -t A ${HOST} 2>/dev/null | grep -o 'has address .*' | awk '{print $3}' | head -n1`
      fi
    fi
    if [ -z "${IP}" ] ; then
      echo "Unable to resolve ${HOST}" >&2
    else
      [ -z "${PORT}" ]    &&    PORT=${PORT_DEFAULT}      # port of Freetz interface
      [ -z "${PASS}" ]    &&    PASS=${PASS_DEFAULT}      # internal password
      [ -z "${USER}" ]    &&    USER=${USER_DEFAULT}      # external username
      [ -z "${PASSE}" ]   &&   PASSE=${PASS}              # external password
      [ -z "${PORTAVM}" ] && PORTAVM=${PORTAVM_DEFAULT}   # port of AVM interface (https)

      DEST_FOLDER="${FOLDER}/${HOST}"
      DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}"

      if [ ${PORT} -eq 0 ] ; then
        get_avm_config     # Fetch only AVM-config when the Freetz port is defined as 0
      else
        get_both_configs
      fi
    fi
  fi
done<"${CONFIGFILE}"
rm -f "${CONFIGFILE}"
 
Zuletzt bearbeitet:
Great work! I'll try the new version in the next days.

I would like to see your updates here :)

BTW:
I would like to see a commandline for the list of modems since i'm using .sh on my shellscripts at the end and so my list would be named freetz_getconfigs.sh.list ;)
 
@vice_pres...

I did some modifications over the weekend and I'm considering the script stable and full featured now.
I also added some code to get rid of the suffix (in your case .sh), but I just re-read your post and you wanted it in there...

A bit strange as it was using a "freetz_getconfigs.sh.list" in the old version of the script if the name of the script was "freetz_getconfigs.sh"

I now especially added code to get rid of the suffix if one exists and am going to leave it that way as it's much neater...

I tested the script to get the AVM-config from LAN-side. It works!
It needs to use http instead of https and it does this by looking at the port. If it starts with an "8" it will use http instead of https.

I enhanced the method to better determine a changed config. Dynamic data like phonebooks are deliberately being ignored.
I would get too many configs if I didn't (backups should be done automatically).

I'm keeping the oldest config, because you will then also have the info when that config has been changed.
The downside of this approach is that you may not have the latest phonebook data in your config.....

Maybe I will modify the script (by request) if someone would prefer to keep the newest config (and kill the older similar one)
 
Zuletzt bearbeitet:
Just a quick note - after upgrading my Ubuntu to 14.04 the script doesn't work because curl changed it behaviour. I had to add --cipher RC4-SHA to every curl call to get the script working again.
 
Thanks vice_pres for this feedback!

I have modified the code based on your findings.
I do make a difference between https / http connections (in case the program is run from the inside of the Fritzbox).
No idea what curl does when I supply these options on a plain http, but it is always better to keep code clean and supply parameters as they are expected... (garbage in => garbage out).

Can you confirm this code is working at your end?

~# cat /usr/local/sbin/freetz_getconfigs
Code:
#!/bin/sh

# Define constants
DATESTAMP=`date +%Y-%m-%d.%H-%M`
FOLDER=/var/www/vhosts/freetz.yourdomain.com/httpdocs

USER_DEFAULT=admin
PASS_DEFAULT="defaultpass"
PORT_DEFAULT=61080
PORTAVM_DEFAULT=450
MINSIZE=20000

HTTP_OPTIONS=
HTTPS_OPTIONS='--cipher RC4-SHA'

HEADLESS=
tty >/dev/null || HEADLESS=true

while getopts cd: opt
do
  case $opt in
    c)   FOLDER="`pwd`" ;;
    d)   FOLDER="$OPTARG";;
    ?)   printf "Usage: %s [-c] [-d <folder>] [<HOST>],[<IP>],[PASS_FREETZ],[<PORT_FREETZ>],[<USER>],[PASS_EXTERNAL],[PORT_AVM]\n" $0
    exit 2;;
  esac
done
shift `expr ${OPTIND} - 1`


# Subroutines

age_of_file ()
{
  if [ ! -e "$1" ] ; then
    echo 0
  else
    AGE_IN_SECONDS=$((`date +%s` - `date +%s -r "$1"` ))
    echo $((${AGE_IN_SECONDS} / 86400))
  fi
}

httpspeed ()
{
  # Returns no value and  errorlevel 1 if site is unreachable or too slow
  # Returns the speed in milliseconds if it's below 2 seconds
  _SPEED=`httping -c1 -t1 ${1} 2>/dev/null | \
         egrep '.+/.+/.+/.+/.+' | tail -n1 | \
         awk -F/ '{print $4}' | awk -F. '{print $1}'`

  if [ -z "${_SPEED}" ] ; then
    echo -e "${URL} is unreachable\n" >&2
  elif [ ${_SPEED} -gt 2000 ] ; then
    echo "${URL} is slow (${_SPEED} ms)" >&2
  else
    echo ${_SPEED}
    return 0
  fi
  return 1
}


get_avm_config ()
{
  [ ${PORTAVM} -eq 0 ] && return

  TMPDIR2=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
  if ! cd ${TMPDIR2} 2>/dev/null ; then
    echo "Error changing current directory to ${TMPDIR2}. You should never see this message!" >&2
  else

    # It's intended to be called from WAN (https)
    # If PORTAVM begins with 8 it assumes LAN and turns to http
    PROTO=https
    echo ${PORTAVM} | grep -q '^80$' && PROTO=http

    OPTIONS="${HTTP_OPTIONS}"
    [ "${PROTO}" = 'https' ] && OPTIONS="${HTTPS_OPTIONS}"

    URL="${PROTO}://${IP}:${PORTAVM}"

    if httpspeed ${URL} >/dev/null ; then
      # get challenge key from FB

      _CHALLENGE=`curl -s -k ${OPTIONS} \
                       --user ${USER}:${PASSE} \
                       "${URL}/login.lua" | \
                       grep 'challenge' | egrep '[0-9a-f]{8}' | \
                       awk -F= '{print $2}' | tail -n1 | egrep -o '[0-9a-f]{8}'`

      if [ -z "${_CHALLENGE}" ] ; then
        echo "Error getting challenge for ${HOST} using ${URL}" >&2
      else
        # echo "Received Challenge \"${_CHALLENGE}\"" >&2
        # build md5 from challenge key and password
        _MD5=`echo -n ${_CHALLENGE}"-"${PASSE} | \
              iconv -f ISO8859-1 -t UTF-16LE   | \
              md5sum -b | awk '{print substr($0,1,32)}'`

        # assemble challenge key and md5
        _RESPONSE=${_CHALLENGE}"-"${_MD5}

        # get sid for later use
        _SID=`curl -i -s -k ${OPTIONS} \
                    --user ${USER}:${PASSE} \
                    -d 'response='${_RESPONSE} \
                    -d 'page=' \
                    -d 'username='${USER} \
                    ${URL}/login.lua | \
                    grep "Location:" | awk -F'=' {' print $NF '}`

        if ! curl -s -k ${OPTIONS} \
                  --user ${USER}:${PASSE} \
                  --form 'sid='${_SID} \
                  --form 'ImportExportPassword='${PASSE} \
                  --form 'ConfigExport=' \
                  ${URL}/cgi-bin/firmwarecfg >TMP1 ; then

          echo "Error getting .export for ${HOST} (no data)" >&2
        else
          if [ `stat -c%s TMP1` -lt ${MINSIZE} ] ; then
            echo "Error getting .export for ${HOST} using ${URL} (too small)" >&2
            echo "SID: \"${_SID}\" CHALLENGE: \"${_CHALLENGE}\"  MD5: \"${_MD5}\"  RESPONSE: \"${_RESPONSE}\"" >&2
          else
            chown ${FUSER}:${FGROUP} TMP1
            mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
            LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
                          grep "${IP}.*\.export" | xargs -I{} stat -c '%Y %n' {} | \
                          sort -rn | head -n1 | awk '{print $2}'`

            if [ -z "${LATESTCONFIG}" ] ; then                     # No previous .export is found
              echo "Saving \"${DEST_BASE}.export\" because no previous AVM-config has been found" >&2
              mv TMP1 "${DEST_BASE}.export"
            else
              if [ ${TRIED_FREETZ} ] ; then      # The Freetz comparison is better, so make that leading
                if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
                  echo -e "Saving \"${DEST_BASE}.export\" because Freetz config has changed\n" >&2
                  mv TMP1 "${DEST_BASE}.export"
                fi
              else
                if [ `age_of_file "${LATESTCONFIG}"` -gt 30 ] ; then      # Compare EXACTLY if age is older than 30 days
                  if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
                    echo "Saving \"${DEST_BASE}.export\" because latest AVM-config was older than 30 days and is not EXACTLY the same" >&2
                    mv TMP1 "${DEST_BASE}.export"
                  fi
                else

                  # remove BINARY data and timestamps before making a diff
                  sed    '/ BINFILE:/,/ END OF FILE /d' "${LATESTCONFIG}" >TMP2
                  sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d'            TMP2
                  sed -i '/END OF EXPORT/d;/update_found/d'                TMP2
                  sed -i '/END OF EXPORT/d;/running_version/d'             TMP2

                  sed    '/ BINFILE:/,/ END OF FILE /d'             TMP1  >TMP3
                  sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d'            TMP3
                  sed -i '/END OF EXPORT/d;/update_found/d'                TMP3
                  sed -i '/END OF EXPORT/d;/running_version/d'             TMP3

                  if ! diff TMP2 TMP3 >/dev/null 2>&1 ; then                  # Compare the 2 stripped configs
                    echo -e "Saving \"${DEST_BASE}.export\" because AVM-config has changed\n" >&2
                    diff TMP2 TMP3 >&2
                    echo -e '\n---\n\n' >&2
                    mv TMP1 "${DEST_BASE}.export"
                  fi
                fi
              fi
            fi
          fi
        fi
      fi
    fi
  fi
  cd "${FOLDER}"
  rm -rf ${TMPDIR2}
}

patch_files ()
{
  # remove or patch some of the untarred files as they spoil the check for a changed config
  if cd "$1" >/dev/null 2>&1 ; then
    rm -f stat.cfg chrony.drift multid.leases 2>/dev/null
    rm -f fx_cg fonctrl phonebook voipd_call_stat 2>/dev/null

    sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' voip.cfg  # remove datestamp in voip.cfg
    sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' ar7.cfg   # remove datestamp in ar7.cfg
    sed -i 's/ *\/\*AVM\*\///g' ar7.cfg                     # remove the /*AVM*/ tag in ar7.cfg
    sed -i 's/; *$/;/g' ar7.cfg                             # remove trailing spaces after a semicolon
    cd - >/dev/null 2>&1
  fi
}

get_freetz_config ()
{
  # Fetch config-file with wget in background
  # Somehow it doesn't listen to the 6 seconds timeout I'm giving it
  wget -q --timeout=6                              \
       --http-user=${USER} --http-password=${PASS} \
       ${URL}/cgi-bin/backup/do_backup.cgi         \
       -O TMP_CONFIG 2>/dev/null &
  # Little wait loop to give wget the time to fetch the file
  n=1
  while sleep 1 ; do
    [ `stat -c%s TMP_CONFIG` -ge ${MINSIZE} ] && break
    [ $n -gt 7 ]                              && break
    let n+=1
  done

  if [ `stat -c%s TMP_CONFIG` -gt 300 ] ; then
    # If I received at least some data, then wait some more and it might get everything
    sleep 2
  else
    echo "wget didn't get any data from host \"${HOST}\"" >&2
    exec 2>/dev/null
    kill -9 %1 2>/dev/null
    exec 2>&2
    return 1
  fi
  return 0
}

keep_config_if_different ()
{
  LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
                      grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | \
                      sort -rn | head -n1 | awk '{print $2}'`

  # If no older config is found
  if [ -z "${LATESTCONFIG}" ] ; then
    echo "Saving \"${DEST_BASE}.tgz\" because no previous Freetz config has been found" >&2
    mv TMP_CONFIG "${DEST_BASE}.tgz"
    get_avm_config
  else

    TRIED_FREETZ=1  # A flag for get_avm_config to know if the freetz config has been tried or not
    mkdir f1   # folder to untar the current config
    mkdir f2   # folder to untar the previous config
    if ! tar xzf TMP_CONFIG -C f1 2>/dev/null ; then
      echo "I just downloaded an invalid gzipped tar-file" >&2
      rm -f TMP_CONFIG
    else
      if ! tar xzf ${LATESTCONFIG} -C f2 ; then
        echo "A previously downloaded config turns out to be invalid... very strange... (${LATESTCONFIG})" >&2
      else
        # Remove files that are likely to be changed, but don't contain important config
        patch_files f1/var_flash
        patch_files f2/var_flash

        if ! diff f1/var_flash f2/var_flash >/dev/null 2>&1 ; then
          echo -e "Saving \"${DEST_BASE}.tgz\" because Freetz config changed\n" >&2
          mv TMP_CONFIG "${DEST_BASE}.tgz"
          # Now get the AVM config too
          get_avm_config
        elif [ `find "${DEST_FOLDER}" -maxdepth 1 -type f -name \*.export | wc -l` -lt 1 ] ; then
          # No previous AVM config exists, so try and fetch one anyhow
          get_avm_config
       fi
     fi
   fi
 fi
}

get_both_configs ()
{
  PROTO=http
  echo ${PORT} | grep -q '^4[456][0-9]' && PROTO=https
  URL="${PROTO}://${IP}:${PORT}"

  if ! httpspeed ${URL} >/dev/null ; then
    # Freetz is not reacting, maybe it is a plain jane AVM Fritzbox
    # I will not be able to see if a config has changed.......
    echo "No reaction from Freetz (\"${HOST}\").... will try AVM now.."
    get_avm_config
  else
    TMPDIR1=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`

    if ! cd ${TMPDIR1} ; then
      echo "Error changing folder to ${TMPDIR1}" >&2
    else
      touch TMP_CONFIG

      get_freetz_config

      if [ `stat -c%s TMP_CONFIG` -lt ${MINSIZE} ] ; then
        echo "Error getting config from ${HOST} on ${IP}" >&2
      else

        # I think we're getting successful here, let's create a folder already
        chown ${FUSER}:${FGROUP} TMP_CONFIG
        mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"

        keep_config_if_different
      fi
    fi
    cd "${FOLDER}"
    rm -rf ${TMPDIR1}
  fi
}


# Main Program starts here

if ! cd "${FOLDER}" ; then
  echo "${FOLDER} does not exist" >&2
  exit 1
fi

BASENAME="${0//*\/}"                           # Strip Path of script name
BASENAME="${BASENAME%.*}"                      # Strip extension
CONFIGSOURCE="${FOLDER}/${BASENAME}.list"      # Construct modemlist from script name
CONFIGFILE=`mktemp`                            # temp file for sanitized modemlist

if [ -n "$*" ] ; then
  echo "$*" >"${CONFIGFILE}"                     # push the commandline parameters in the configfile
elif [ -e "${CONFIGSOURCE}" ] ; then
  sed '/^ *$/d;/^ *#/d' "${CONFIGSOURCE}" | \
  awk -F'[\t ]#' '{print $1}' >"${CONFIGFILE}"             # Get all the data before # and remove empty lines
else
  echo "Configfile: \"${CONFIGSOURCE}\" does not exist" >&2
  exit 1
fi

if [ ! -s "${CONFIGFILE}" ] ; then
  echo "No workable data in \"${CONFIGSOURCE}\"" >&2
  exit 1
fi

# Main Loop
FGROUP=`stat -c%G .`
FUSER=`stat -c%U .`

while read MODEMLINE ; do
  HOST="`echo "${MODEMLINE}"    | awk -F, '{print $1}' | awk '{print $1}'`"
  IP="`echo "${MODEMLINE}"      | awk -F, '{print $2}' | awk '{print $1}'`"
  PASS="`echo "${MODEMLINE}"    | awk -F, '{print $3}' | awk '{print $1}'`"
  PORT="`echo "${MODEMLINE}"    | awk -F, '{print $4}' | tr -cd '0-9'`"
  USER="`echo "${MODEMLINE}"    | awk -F, '{print $5}' | awk '{print $1}'`"
  PASSE="`echo "${MODEMLINE}"   | awk -F, '{print $6}' | awk '{print $1}'`"
  PORTAVM="`echo "${MODEMLINE}" | awk -F, '{print $7}' | tr -cd '0-9'`"

  HOSTORG="${HOST}"
  TRIED_FREETZ=

  if [ -z "${IP}" ] ; then
    HOST=`echo "${HOST}" | tr 'A-Z' 'a-z' | \
          egrep -o '[a-z0-9.-]+\.[a-z0-9]+'`     # Loosely Sanitize domain if no IP is given
  fi
  if [ -z "${HOST}" ] ; then
    echo "${HOSTORG} is not a valid host" >&2
  else
    [ ${HEADLESS} ] || echo ${HOST}
    if [ -z "${IP}" ] ; then
      # If HOST is so clearly an IP-address then use that instead....
      if echo "${HOST}" | egrep -q '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$' ; then
        IP=${HOST}
      else
        IP=`host -t A ${HOST} 2>/dev/null | grep -o 'has address .*' | awk '{print $3}' | head -n1`
      fi
    fi
    if [ -z "${IP}" ] ; then
      echo "Unable to resolve ${HOST}" >&2
    else
      [ -z "${PORT}" ]    &&    PORT=${PORT_DEFAULT}      # port of Freetz interface
      [ -z "${PASS}" ]    &&    PASS=${PASS_DEFAULT}      # internal password
      [ -z "${USER}" ]    &&    USER=${USER_DEFAULT}      # external username
      [ -z "${PASSE}" ]   &&   PASSE=${PASS}              # external password
      [ -z "${PORTAVM}" ] && PORTAVM=${PORTAVM_DEFAULT}   # port of AVM interface (https)

      DEST_FOLDER="${FOLDER}/${HOST}"
      DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}"

      if [ ${PORT} -eq 0 ] ; then
        get_avm_config     # Fetch only AVM-config when the Freetz port is defined as 0
      else
        get_both_configs
      fi
    fi
  fi
done<"${CONFIGFILE}"
rm -f "${CONFIGFILE}"
 
Zuletzt bearbeitet:
Hi,

Code is working, I'm using a version I modified a bit (for example storing hostnames in the filename instead of ip because they are dynamic) etc - but tried your code in a new file and it's working :)

Edit:
I changed something to the sed when comparing the configs since the passwords seem to be rehashed everytime something is changed via webinterface - so there would always be a diff...
sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d'
 
Zuletzt bearbeitet:
I will look into that sed later. I did notice some configs get an update when in fact nothing really changed.

I don't understand your remark about the dynamic host.
My script fully supports dynamic hosts. Just leave out the IP.
 
Yeah, but the script stores the files with the ip-adress - and I find it more usefull if the script stores the hostname instead.

Code:
test.dyndns.org
Saving "/new/test.dyndns.org/83.20.222.212.2014-04-27.16-06.export" because no previous AVM-config has been found

Because you set DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}" and I modified it to be DEST_BASE="${DEST_FOLDER}/${HOST}.${DATESTAMP}" :)
 
Hi,

Since 06.30 the '--cipher RC4-SHA' option is not needed anymore - in fact it stops the backup from working.
 
I will check my script and publish the new one...
I will also check why I have a "new config" each day. Something must have been added that changes every day.
I need to add something more to the ignore part..

Will post something soon
 
I modified the Script to have the HTTPS_Option as an extra option in the config file - since I have some Boxes that don't get 06.30 anymore (7270 for example) and need the cipher. But since I started on an older version of your script a diff is rather long - plus the modifications I wanted (storing in folder named as the hostname etc...). This version works fine for all boxes I'm backing up - the 06.30 Boxes get backed up almost daily (but not daily - some days there are no diffs. I assume it's the call log that changed and triggers an backup)

Code:
1c1
< #!/bin/bash
---
> #!/bin/sh
5c5
< FOLDER=/fritzbox/backup
---
> FOLDER=/var/www/vhosts/freetz.yourdomain.com/httpdocs
8,9c8,9
< PASS_DEFAULT=topsecret
< PORT_DEFAULT=6090
---
> PASS_DEFAULT="defaultpass"
> PORT_DEFAULT=61080
11a12,15
>
> HTTP_OPTIONS=
> HTTPS_OPTIONS='--cipher RC4-SHA'
>
20c24
<     ?)   printf "Usage: %s [-c] [-d <folder>] [<HOST>],[<IP>],[PASS_FREETZ],[<PORT_FREETZ>],[<USER>],[PASS_EXTERNAL],[PORT_AVM],[CURL_OPTS]\n" $0
---
>     ?)   printf "Usage: %s [-c] [-d <folder>] [<HOST>],[<IP>],[PASS_FREETZ],[<PORT_FREETZ>],[<USER>],[PASS_EXTERNAL],[PORT_AVM]\n" $0
48c52
<     echo "${URL} is unreachable" >&2
---
>     echo -e "${URL} is unreachable\n" >&2
71c75,79
<     echo ${PORTAVM} | grep -q '^8' && PROTO=http
---
>     echo ${PORTAVM} | grep -q '^80$' && PROTO=http
>
>     OPTIONS="${HTTP_OPTIONS}"
>     [ "${PROTO}" = 'https' ] && OPTIONS="${HTTPS_OPTIONS}"
>
74c82
<     #if httpspeed ${URL} >/dev/null ; then
---
>     if httpspeed ${URL} >/dev/null ; then
77c85
<       _CHALLENGE=`curl ${CURLOPTS} -s -k \
---
>       _CHALLENGE=`curl -s -k ${OPTIONS} \
96c104
<         _SID=`curl ${CURLOPTS} -i -s -k \
---
>         _SID=`curl -i -s -k ${OPTIONS} \
104c112
<         if ! curl ${CURLOPTS} -s -k \
---
>         if ! curl -s -k ${OPTIONS} \
118c126
<             mkdir -p "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
---
>             mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
120c128
<                           grep "${HOST}.*\.export" | xargs -I{} stat -c '%Y %n' {} | \
---
>                           grep "${IP}.*\.export" | xargs -I{} stat -c '%Y %n' {} | \
129c137
<                   echo "Saving \"${DEST_BASE}.export\" because Freetz config has changed" >&2
---
>                   echo -e "Saving \"${DEST_BASE}.export\" because Freetz config has changed\n" >&2
135c143
<                     echo "Saving \"${DEST_BASE}.export\" because latest AVM-config was older than 30 days and is different" >&2
---
>                     echo "Saving \"${DEST_BASE}.export\" because latest AVM-config was older than 30 days and is not EXACTLY the same" >&2
139,144c147,158
<                   # remove BINARY data, timestamps and passwords (since OS6.0 passwords are rehashed at every configchange through webif it seems) before making a diff
<                   #sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d' "${LATESTCONFIG}" >TMP2
<                   #sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d' TMP1              >TMP3
<                 # just remove timestamps and passwords (since OS6.0 passwords are rehashed at every configchange through webif it seems)
<                   sed '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d' "${LATESTCONFIG}" >TMP2
<                   sed '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d' TMP1              >TMP3
---
>
>                   # remove BINARY data and timestamps before making a diff
>                   sed    '/ BINFILE:/,/ END OF FILE /d' "${LATESTCONFIG}" >TMP2
>                   sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d'            TMP2
>                   sed -i '/END OF EXPORT/d;/update_found/d'                TMP2
>                   sed -i '/END OF EXPORT/d;/running_version/d'             TMP2
>
>                   sed    '/ BINFILE:/,/ END OF FILE /d'             TMP1  >TMP3
>                   sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d'            TMP3
>                   sed -i '/END OF EXPORT/d;/update_found/d'                TMP3
>                   sed -i '/END OF EXPORT/d;/running_version/d'             TMP3
>
146,147c160,162
<                     echo "Saving \"${DEST_BASE}.export\" because AVM-config has changed" >&2
<                     #diff TMP2 TMP3
---
>                     echo -e "Saving \"${DEST_BASE}.export\" because AVM-config has changed\n" >&2
>                     diff TMP2 TMP3 >&2
>                     echo -e '\n---\n\n' >&2
156c171
<     #fi
---
>     fi
209c224
<                       grep "${HOST}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | \
---
>                       grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | \
220,221c235,236
<     mkdir -p f1   # folder to untar the current config
<     mkdir -p f2   # folder to untar the previous config
---
>     mkdir f1   # folder to untar the current config
>     mkdir f2   # folder to untar the previous config
234c249
<           echo "Saving \"${DEST_BASE}.tgz\" because Freetz config changed" >&2
---
>           echo -e "Saving \"${DEST_BASE}.tgz\" because Freetz config changed\n" >&2
256c271
<     echo "No reaction from Freetz.... will try AVM now.."
---
>     echo "No reaction from Freetz (\"${HOST}\").... will try AVM now.."
274c289
<         mkdir -p "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
---
>         mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
300,301c315,316
<   awk -F# '{print $1}' "${CONFIGSOURCE}" | \
<       sed '/^ *$/d' >"${CONFIGFILE}"             # Get all the data before # and remove empty lines
---
>   sed '/^ *$/d;/^ *#/d' "${CONFIGSOURCE}" | \
>   awk -F'[\t ]#' '{print $1}' >"${CONFIGFILE}"             # Get all the data before # and remove empty lines
317,319c332,334
<   HOST="`echo "${MODEMLINE}"    | awk -F, '{print $1}'`"
<   IP="`echo "${MODEMLINE}"      | awk -F, '{print $2}'`"
<   PASS="`echo "${MODEMLINE}"    | awk -F, '{print $3}'`"
---
>   HOST="`echo "${MODEMLINE}"    | awk -F, '{print $1}' | awk '{print $1}'`"
>   IP="`echo "${MODEMLINE}"      | awk -F, '{print $2}' | awk '{print $1}'`"
>   PASS="`echo "${MODEMLINE}"    | awk -F, '{print $3}' | awk '{print $1}'`"
321,322c336,337
<   USER="`echo "${MODEMLINE}"    | awk -F, '{print $5}'`"
<   PASSE="`echo "${MODEMLINE}"   | awk -F, '{print $6}'`"
---
>   USER="`echo "${MODEMLINE}"    | awk -F, '{print $5}' | awk '{print $1}'`"
>   PASSE="`echo "${MODEMLINE}"   | awk -F, '{print $6}' | awk '{print $1}'`"
324d338
<   CURLOPTS="`echo "${MODEMLINE}"   | awk -F, '{print $8}'`"
355c369
<       DEST_BASE="${DEST_FOLDER}/${HOST}.${DATESTAMP}"
---
>       DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}"
 
My script is supposed to ignore the call logs as far as I can remember... If I have time I will take a look.
I haven't noticed that I wasn't getting any configs of some routers...
Didn't find the time to check. Recently bought my first 3d printer
 
Just as a warning: The script can't backup 06.50 at the moment as it seems - and reboots the target (when running 06.50) after trying to get a backup. I'll try if I can take a look why this is happening on monday using my lab...

This is working from 06.06 (7270 V2) to 7490 (06.50) on all verisons I can test:

Code:
      _CHALLENGE=`curl ${CURLOPTS} -s -k \
                       --user ${USER}:${PASSE} \
                       "${URL}/login_sid.lua" | \
                       grep Challenge | sed -e 's/.*<Challenge>//' | sed -e 's/<\/Challenge>.*//'`

Code:
        _SID=`curl ${CURLOPTS} -i -s -k \
                    --user ${USER}:${PASSE} \
                    -d 'response='${_RESPONSE} \
                    -d 'page=' \
                    -d 'username='${USER} \
                    ${URL}/login_sid.lua | \
                    grep SID | sed -e 's/.*<SID>//' | sed -e 's/<\/SID>.*//'`

Since there is no login.lua anymore I changed it to login_sid.lua which is available on all 06 Versions I can test. But my testbox doesn't reboot like my production box when testing...
 
Zuletzt bearbeitet:
I am very very late implementing your fixes due to all kind of other stuff...
Here's the complete script again implementing your fixes.....

I don't know if it still can download the export of old fritzboxes....
The box on which I was testing it now works.....

# cat /usr/local/sbin/freetz_getconfigs
Code:
#!/bin/sh


# Define constants
DATESTAMP=`date +%Y-%m-%d.%H-%M`
FOLDER=/var/www/vhosts/freetz.mr-wolf.nl/httpdocs/courant


USER_DEFAULT=admin
PASS_DEFAULT=q1w2e3r4
PORT_DEFAULT=61080
PORTAVM_DEFAULT=450
MINSIZE=20000


HTTP_OPTIONS=''
HTTPS_OPTIONS=''
# HTTPS_OPTIONS='--cipher RC4-SHA'


HEADLESS=
tty >/dev/null || HEADLESS=true


while getopts cd: opt
do
  case $opt in
    c)   FOLDER="`pwd`" ;;
    d)   FOLDER="$OPTARG";;
    ?)   printf "Usage: %s [-c] [-d <folder>] [<HOST>],[<IP>],[PASS_FREETZ],[<PORT_FREETZ>],[<USER>],[PASS_EXTERNAL],[PORT_AVM]\n" $0
    exit 2;;
  esac
done
shift `expr ${OPTIND} - 1`




# Subroutines


age_of_file ()
{
  if [ ! -e "$1" ] ; then
    echo 0
  else
    AGE_IN_SECONDS=$((`date +%s` - `date +%s -r "$1"` ))
    echo $((${AGE_IN_SECONDS} / 86400))
  fi
}


httpspeed ()
{
  # Returns no value and  errorlevel 1 if site is unreachable or too slow
  # Returns the speed in milliseconds if it's below 2 seconds
  _SPEED=`httping -c1 -t1 ${1} 2>/dev/null | \
         egrep '.+/.+/.+/.+/.+' | tail -n1 | \
         awk -F/ '{print $4}' | awk -F. '{print $1}'`


  if [ -z "${_SPEED}" ] ; then
    echo -e "${URL} is unreachable\n" >&2
  elif [ ${_SPEED} -gt 2000 ] ; then
    echo "${URL} is slow (${_SPEED} ms)" >&2
  else
    echo ${_SPEED}
    return 0
  fi
  return 1
}




get_avm_config ()
{
  [ ${PORTAVM} -eq 0 ] && return


  TMPDIR2=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
  if ! cd ${TMPDIR2} 2>/dev/null ; then
    echo "Error changing current directory to ${TMPDIR2}. You should never see this message!" >&2
  else


    # It's intended to be called from WAN (https)
    # If PORTAVM begins with 8 it assumes LAN and turns to http
    PROTO=https
    echo ${PORTAVM} | grep -q '^80$' && PROTO=http


    CURLOPTS="${HTTP_OPTIONS}"
    [ "${PROTO}" = 'https' ] && CURLOPTS="${HTTPS_OPTIONS}"


    URL="${PROTO}://${IP}:${PORTAVM}"


    if httpspeed ${URL} >/dev/null ; then
      # get challenge key from FB


      _CHALLENGE=`curl ${CURLOPTS} -s -k \
                       --user ${USER}:${PASSE} \
                       "${URL}/login_sid.lua" | \
                       grep 'Challenge' | sed -e 's/.*<Challenge>//' | sed -e 's/<\/Challenge>.*//'`


      if [ -z "${_CHALLENGE}" ] ; then
        echo "Error getting challenge for ${HOST} using ${URL}/login_sid.lua" >&2
      else
        # echo "Received Challenge \"${_CHALLENGE}\"" >&2
        # build md5 from challenge key and password
        _MD5=`echo -n ${_CHALLENGE}"-"${PASSE} | \
              iconv -f ISO8859-1 -t UTF-16LE   | \
              md5sum -b | awk '{print substr($0,1,32)}'`


        # assemble challenge key and md5
        _RESPONSE=${_CHALLENGE}"-"${_MD5}


        # get sid for later use
        _SID=`curl ${CURLOPTS} -i -s -k \
                    --user ${USER}:${PASSE} \
                    -d 'response='${_RESPONSE} \
                    -d 'page=' \
                    -d 'username='${USER} \
                    ${URL}/login_sid.lua | \
                    grep SID | sed -e 's/.*<SID>//' | sed -e 's/<\/SID>.*//'`


        if ! curl ${CURLOPTS} -s -k \
                  --user ${USER}:${PASSE} \
                  --form 'sid='${_SID} \
                  --form 'ImportExportPassword='${PASSE} \
                  --form 'ConfigExport=' \
                  ${URL}/cgi-bin/firmwarecfg >TMP1 ; then


          echo "Error getting .export for ${HOST} (no data)" >&2
        else
          if [ `stat -c%s TMP1` -lt ${MINSIZE} ] ; then
            echo "Error getting .export for ${HOST} using ${URL} (too small)" >&2
            echo "SID: \"${_SID}\" CHALLENGE: \"${_CHALLENGE}\"  MD5: \"${_MD5}\"  RESPONSE: \"${_RESPONSE}\"" >&2
          else
            chown ${FUSER}:${FGROUP} TMP1
            mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
            LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
                          grep "${IP}.*\.export" | xargs -I{} stat -c '%Y %n' {} | \
                          sort -rn | head -n1 | awk '{print $2}'`


            if [ -z "${LATESTCONFIG}" ] ; then                     # No previous .export is found
              echo "Saving \"${DEST_BASE}.export\" because no previous AVM-config has been found" >&2
              mv TMP1 "${DEST_BASE}.export"
            else
              if [ ${TRIED_FREETZ} ] ; then      # The Freetz comparison is better, so make that leading
                if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
                  echo -e "Saving \"${DEST_BASE}.export\" because Freetz config has changed\n" >&2
                  mv TMP1 "${DEST_BASE}.export"
                fi
              else
                if [ `age_of_file "${LATESTCONFIG}"` -gt 30 ] ; then      # Compare EXACTLY if age is older than 30 days
                  if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
                    echo "Saving \"${DEST_BASE}.export\" because latest AVM-config was older than 30 days and is not EXACTLY the same" >&2
                    mv TMP1 "${DEST_BASE}.export"
                  fi
                else


                  # remove BINARY data and timestamps before making a diff
                  sed    '/ BINFILE:/,/ END OF FILE /d' "${LATESTCONFIG}" >TMP2
                  sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d'            TMP2
                  sed -i '/END OF EXPORT/d;/update_found/d'                TMP2
                  sed -i '/\$\$\$\$/d;/running_version/d'                  TMP2


                  sed    '/ BINFILE:/,/ END OF FILE /d'             TMP1  >TMP3
                  sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d'            TMP3
                  sed -i '/END OF EXPORT/d;/update_found/d'                TMP3
                  sed -i '/\$\$\$\$/d;/running_version/d'                  TMP3


                  if ! diff TMP2 TMP3 >/dev/null 2>&1 ; then                  # Compare the 2 stripped configs
                    echo -e "Saving \"${DEST_BASE}.export\" because AVM-config has changed\n" >&2
                    diff TMP2 TMP3 >&2
                    echo -e '\n---\n\n' >&2
                    mv TMP1 "${DEST_BASE}.export"
                  fi
                fi
              fi
            fi
          fi
        fi
      fi
    fi
  fi
  cd "${FOLDER}"
  rm -rf ${TMPDIR2}
}


patch_files ()
{
  # remove or patch some of the untarred files as they spoil the check for a changed config
  if cd "$1" >/dev/null 2>&1 ; then
    rm -f userstat.cfg stat.cfg chrony.drift multid.leases 2>/dev/null
    rm -f fx_cg fonctrl phonebook voipd_call_stat 2>/dev/null


    sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' voip.cfg  # remove datestamp in voip.cfg
    sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' ar7.cfg   # remove datestamp in ar7.cfg
    sed -i 's/ *\/\*AVM\*\///g' ar7.cfg                     # remove the /*AVM*/ tag in ar7.cfg
    sed -i 's/; *$/;/g' ar7.cfg                             # remove trailing spaces after a semicolon
    cd - >/dev/null 2>&1
  fi
}


get_freetz_config ()
{
  # Fetch config-file with wget in background
  # Somehow it doesn't listen to the 6 seconds timeout I'm giving it
  wget -q --timeout=6                              \
       --http-user=${USER} --http-password=${PASS} \
       ${URL}/cgi-bin/backup/do_backup.cgi         \
       -O TMP_CONFIG 2>/dev/null &
  # Little wait loop to give wget the time to fetch the file
  n=1
  while sleep 1 ; do
    [ `stat -c%s TMP_CONFIG` -ge ${MINSIZE} ] && break
    [ $n -gt 7 ]                              && break
    let n+=1
  done


  if [ `stat -c%s TMP_CONFIG` -gt 300 ] ; then
    # If I received at least some data, then wait some more and it might get everything
    sleep 2
  else
    echo "wget didn't get any data from host \"${HOST}\"" >&2
    exec 2>/dev/null
    kill -9 %1 2>/dev/null
    exec 2>&2
    return 1
  fi
  return 0
}


keep_config_if_different ()
{
  LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
                      grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | \
                      sort -rn | head -n1 | awk '{print $2}'`


  # If no older config is found
  if [ -z "${LATESTCONFIG}" ] ; then
    echo "Saving \"${DEST_BASE}.tgz\" because no previous Freetz config has been found" >&2
    mv TMP_CONFIG "${DEST_BASE}.tgz"
    get_avm_config
  else


    TRIED_FREETZ=1  # A flag for get_avm_config to know if the freetz config has been tried or not
    mkdir f1   # folder to untar the current config
    mkdir f2   # folder to untar the previous config
    if ! tar xzf TMP_CONFIG -C f1 2>/dev/null ; then
      echo "I just downloaded an invalid gzipped tar-file" >&2
      rm -f TMP_CONFIG
    else
      if ! tar xzf ${LATESTCONFIG} -C f2 ; then
        echo "A previously downloaded config turns out to be invalid... very strange... (${LATESTCONFIG})" >&2
      else
        # Remove files that are likely to be changed, but don't contain important config
        patch_files f1/var_flash
        patch_files f2/var_flash


        if ! diff f1/var_flash f2/var_flash >/dev/null 2>&1 ; then
          [ ${HEADLESS} ] || diff f1/var_flash f2/var_flash >&2
          echo -e "Saving \"${DEST_BASE}.tgz\" because Freetz config changed\n" >&2
          mv TMP_CONFIG "${DEST_BASE}.tgz"
          # Now get the AVM config too
          get_avm_config
        elif [ `find "${DEST_FOLDER}" -maxdepth 1 -type f -name \*.export | wc -l` -lt 1 ] ; then
          # No previous AVM config exists, so try and fetch one anyhow
          get_avm_config
       fi
     fi
   fi
 fi
}


get_both_configs ()
{
  PROTO=http
  echo ${PORT} | grep -q '^4[456][0-9]' && PROTO=https
  URL="${PROTO}://${IP}:${PORT}"


  if ! httpspeed ${URL} >/dev/null ; then
    # Freetz is not reacting, maybe it is a plain jane AVM Fritzbox
    # I will not be able to see if a config has changed.......
    echo "No reaction from Freetz (\"${HOST}\").... will try AVM now.."
    get_avm_config
  else
    TMPDIR1=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`


    if ! cd ${TMPDIR1} ; then
      echo "Error changing folder to ${TMPDIR1}" >&2
    else
      touch TMP_CONFIG


      get_freetz_config


      if [ `stat -c%s TMP_CONFIG` -lt ${MINSIZE} ] ; then
        echo "Error getting config from ${HOST} on ${IP}" >&2
      else


        # I think we're getting successful here, let's create a folder already
        chown ${FUSER}:${FGROUP} TMP_CONFIG
        mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"


        keep_config_if_different
      fi
    fi
    cd "${FOLDER}"
    rm -rf ${TMPDIR1}
  fi
}




# Main Program starts here


if ! cd "${FOLDER}" ; then
  echo "${FOLDER} does not exist" >&2
  exit 1
fi


BASENAME="${0//*\/}"                           # Strip Path of script name
BASENAME="${BASENAME%.*}"                      # Strip extension
CONFIGSOURCE="${FOLDER}/${BASENAME}.list"      # Construct modemlist from script name
CONFIGFILE=`mktemp`                            # temp file for sanitized modemlist


if [ -n "$*" ] ; then
  echo "$*" >"${CONFIGFILE}"                     # push the commandline parameters in the configfile
elif [ -e "${CONFIGSOURCE}" ] ; then
  sed '/^ *$/d;/^ *#/d' "${CONFIGSOURCE}" | \
  awk -F'[\t ]#' '{print $1}' >"${CONFIGFILE}"             # Get all the data before # and remove empty lines
else
  echo "Configfile: \"${CONFIGSOURCE}\" does not exist" >&2
  exit 1
fi


if [ ! -s "${CONFIGFILE}" ] ; then
  echo "No workable data in \"${CONFIGSOURCE}\"" >&2
  exit 1
fi


# Main Loop
FGROUP=`stat -c%G .`
FUSER=`stat -c%U .`


while read MODEMLINE ; do
  HOST="`echo "${MODEMLINE}"    | awk -F, '{print $1}' | awk '{print $1}'`"
  IP="`echo "${MODEMLINE}"      | awk -F, '{print $2}' | awk '{print $1}'`"
  PASS="`echo "${MODEMLINE}"    | awk -F, '{print $3}' | awk '{print $1}'`"
  PORT="`echo "${MODEMLINE}"    | awk -F, '{print $4}' | tr -cd '0-9'`"
  USER="`echo "${MODEMLINE}"    | awk -F, '{print $5}' | awk '{print $1}'`"
  PASSE="`echo "${MODEMLINE}"   | awk -F, '{print $6}' | awk '{print $1}'`"
  PORTAVM="`echo "${MODEMLINE}" | awk -F, '{print $7}' | tr -cd '0-9'`"


  HOSTORG="${HOST}"
  TRIED_FREETZ=


  if [ -z "${IP}" ] ; then
    HOST=`echo "${HOST}" | tr 'A-Z' 'a-z' | \
          egrep -o '[a-z0-9.-]+\.[a-z0-9]+'`     # Loosely Sanitize domain if no IP is given
  fi
  if [ -z "${HOST}" ] ; then
    echo "${HOSTORG} is not a valid host" >&2
  else
    [ ${HEADLESS} ] || echo ${HOST}
    if [ -z "${IP}" ] ; then
      # If HOST is so clearly an IP-address then use that instead....
      if echo "${HOST}" | egrep -q '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$' ; then
        IP=${HOST}
      else
        IP=`host -t A ${HOST} 2>/dev/null | grep -o 'has address .*' | awk '{print $3}' | head -n1`
      fi
    fi
    if [ -z "${IP}" ] ; then
      echo "Unable to resolve ${HOST}" >&2
    else
      [ -z "${PORT}" ]    &&    PORT=${PORT_DEFAULT}      # port of Freetz interface
      [ -z "${PASS}" ]    &&    PASS=${PASS_DEFAULT}      # internal password
      [ -z "${USER}" ]    &&    USER=${USER_DEFAULT}      # external username
      [ -z "${PASSE}" ]   &&   PASSE=${PASS}              # external password
      [ -z "${PORTAVM}" ] && PORTAVM=${PORTAVM_DEFAULT}   # port of AVM interface (https)


      DEST_FOLDER="${FOLDER}/${HOST}"
      DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}"


      if [ ${PORT} -eq 0 ] ; then
        get_avm_config     # Fetch only AVM-config when the Freetz port is defined as 0
      else
        get_both_configs
      fi
    fi
  fi
done<"${CONFIGFILE}"
rm -f "${CONFIGFILE}"
 
The download of AVM's config does not work with 6.83 anymore...
Anyone has a quick fix?
Somehow the challenge fails....

EDIT...
The root cause must lie somewhere else.
The script works fine on another 7490 with 6.83 (a higher version, but I will test some more)

EDIT
I found the root cause
It's that two factor challenge method that's spoiling it (where one needs to press a button).
I just tried a manual backup and that failed for these same reasons.

# Edit ar7.cfg and change two_factor_auth_enabled
nvi /var/flash/ar7.cfg

two_factor_auth_enabled = no;

# Stop ctlmgr
ctlmgr -s

# Start ctlmgr
ctlmgr
 
Zuletzt bearbeitet:
Holen Sie sich 3CX - völlig kostenlos!
Verbinden Sie Ihr Team und Ihre Kunden Telefonie Livechat Videokonferenzen

Gehostet oder selbst-verwaltet. Für bis zu 10 Nutzer dauerhaft kostenlos. Keine Kreditkartendetails erforderlich. Ohne Risiko testen.

3CX
Für diese E-Mail-Adresse besteht bereits ein 3CX-Konto. Sie werden zum Kundenportal weitergeleitet, wo Sie sich anmelden oder Ihr Passwort zurücksetzen können, falls Sie dieses vergessen haben.