.titleBar { margin-bottom: 5px!important; }

Automatic backup of config using a "pull method"

Dieses Thema im Forum "Freetz" wurde erstellt von frater, 11 Mai 2013.

  1. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    404
    Zustimmungen:
    1
    Punkte für Erfolge:
    18
    I will improve it in the near future after I have it restructured. It's then also easier to implement little changes
     
  2. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    404
    Zustimmungen:
    1
    Punkte für Erfolge:
    18
    #22 frater, 18 Mai 2013
    Zuletzt bearbeitet: 20 Mai 2013
    I have restructured the script and tested it against the cluster of modems I am supervising.

    It relies on 'httping' to be installed


    The script can take command line parameters now...
    • -c use current directory instead of default
    • -d <folder> use <folder> instead of default
    • comma delimited parameters for 1 modem (same syntax as the list)

    If no parameters for a modem are given, it will search for a list containing modem parameters in the designated folder and parse that list and fetch both the Freetz config and the AVM-config.
    The parameters for 1 modem can be as basic as its external address. It will then use the default values for the rest....

    The AVM-config is password-protected with the external password.
    The files will be placed in a folder and get a little check. If the config is (about) the same as the last one fetched, it will get discarded.
    This way you can just create a symlink to the script in /etc/cron.daily and you will always have a folder with different configs.
    You can easily see when a config has been changed.

    I'm placing those files in a password protected website where I can immediately download one of the configs I need....
    Maybe it will save my ass one day...

    It can now also get the AVM-config if there's no freetz on it.
    If it doesn't detect the Freetz interface it will do an AVM only.
    If you don't want to even try the Freetz-port you can define it as port 0


    The script could do with some better logging and maybe there are some bugs there.

    If you use the script I would appreciate some feedback.
    I also would like to know if I should bother posting updates here.
    Thanks Daniel and MaxMuster for the URL's I needed to get..


    Update May 18 17:30
    Added better handling for "#"
    You can put comments at the end of the modemline and of course in the beginning
    Everything behind the # will be ignored...

    Also added some code to prevent duplicate AVM-configs (in case Freetz is not used)

    Update May 18 00:00
    Not only remove the path, but also the extension from the name of the script to use as the name for modemlist

    Update May 19 12:00
    I should spend my holidays better, really...
    Another improvement to make the detection for changes a bit better and more sensible.
    If only an AVM-config is made, the script kept the config too often. I now check for big differences by ignoring the binary data in the config.
    If, however, the latest config is older than 30 days the script does an exact comparison.....

    Update May 20 13:15
    The script still made some unnecessary backups, so I investigated it and improved the comparison. Added some extra logging...
    If used as a cronjob, it will only output data if a changes is found....

    Some examples:

    Simple help
    Code:
    freetz_getconfigs -?
    Get config in default folder using a list called freetz_getconfigs.list containing all the modems
    Code:
    freetz_getconfigs
    Get config in folder /opt from host remote.domain.com using all default values
    Code:
    freetz_getconfigs -d /opt remote.domain.com
    Get config in current folder from host remote.domain.com using all default values
    Code:
    freetz_getconfigs -c  remote.domain.com
    Get config in current folder from 192.168.178.1 using port 81 for Freetz interface and port 451 for the AVM interface
    Code:
    freetz_getconfigs -c  mymodem,192.168.178.1,,81,,,80
    Get config in current folder from 192.168.178.1 from LAN using a plain Jane AVM Fritzbox without Freetz
    Code:
    freetz_getconfigs -c  mymodem,192.168.178.1,,0,,,80
    Get config in current folder from host remote.domain.com using port 451 for the AVM interface
    Code:
    freetz_getconfigs -c  remote.domain.com,,,,,,451

    # cat /usr/local/sbin/freetz_getconfigs
    Code:
    #!/bin/sh
    
    # Define constants
    DATESTAMP=`date +%Y-%m-%d.%H-%M`
    FOLDER=/var/www/vhosts/domain.com/freetz
    
    USER_DEFAULT=admin
    PASS_DEFAULT=secret
    PORT_DEFAULT=6080
    PORTAVM_DEFAULT=450
    MINSIZE=20000
    HEADLESS=
    tty >/dev/null || HEADLESS=true
    
    while getopts cd: opt
    do
      case $opt in
        c)   FOLDER="`pwd`" ;;
        d)   FOLDER="$OPTARG";;
        ?)   printf "Usage: %s [-c] [-d <folder>] [<HOST>],[<IP>],[PASS_FREETZ],[<PORT_FREETZ>],[<USER>],[PASS_EXTERNAL],[PORT_AVM]\n" $0
        exit 2;;
      esac
    done
    shift `expr ${OPTIND} - 1`
    
    
    # Subroutines
    
    age_of_file ()
    {
      if [ ! -e "$1" ] ; then
        echo 0
      else
        AGE_IN_SECONDS=$((`date +%s` - `date +%s -r "$1"` ))
        echo $((${AGE_IN_SECONDS} / 86400))
      fi
    }
    
    httpspeed ()
    {
      # Returns no value and  errorlevel 1 if site is unreachable or too slow
      # Returns the speed in milliseconds if it's below 2 seconds
      _SPEED=`httping -c1 -t1 ${1} 2>/dev/null | \
             egrep '.+/.+/.+/.+/.+' | tail -n1 | \
             awk -F/ '{print $4}' | awk -F. '{print $1}'`
    
      if [ -z "${_SPEED}" ] ; then
        echo "${URL} is unreachable" >&2
      elif [ ${_SPEED} -gt 2000 ] ; then
        echo "${URL} is slow (${_SPEED} ms)" >&2
      else
        echo ${_SPEED}
        return 0
      fi
      return 1
    }
    
    
    get_avm_config ()
    {
      [ ${PORTAVM} -eq 0 ] && return
    
      TMPDIR2=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
      if ! cd ${TMPDIR2} 2>/dev/null ; then
        echo "Error changing current directory to ${TMPDIR2}. You should never see this message!" >&2
      else
    
        # It's intended to be called from WAN (https)
        # If PORTAVM begins with 8 it assumes LAN and turns to http
        PROTO=https
        echo ${PORTAVM} | grep -q '^8' && PROTO=http
        URL="${PROTO}://${IP}:${PORTAVM}"
    
        if httpspeed ${URL} >/dev/null ; then
          # get challenge key from FB
    
          _CHALLENGE=`curl -s -k \
                           --user ${USER}:${PASSE} \
                           "${URL}/login.lua" | \
                           grep 'challenge' | egrep '[0-9a-f]{8}' | \
                           awk -F= '{print $2}' | tail -n1 | egrep -o '[0-9a-f]{8}'`
    
          if [ -z "${_CHALLENGE}" ] ; then
            echo "Error getting challenge for ${HOST} using ${URL}" >&2
          else
            # echo "Received Challenge \"${_CHALLENGE}\"" >&2
            # build md5 from challenge key and password
            _MD5=`echo -n ${_CHALLENGE}"-"${PASSE} | \
                  iconv -f ISO8859-1 -t UTF-16LE   | \
                  md5sum -b | awk '{print substr($0,1,32)}'`
    
            # assemble challenge key and md5
            _RESPONSE=${_CHALLENGE}"-"${_MD5}
    
            # get sid for later use
            _SID=`curl -i -s -k \
                        --user ${USER}:${PASSE} \
                        -d 'response='${_RESPONSE} \
                        -d 'page=' \
                        -d 'username='${USER} \
                        ${URL}/login.lua | \
                        grep "Location:" | awk -F'=' {' print $NF '}`
    
            if ! curl -s -k \
                      --user ${USER}:${PASSE} \
                      --form 'sid='${_SID} \
                      --form 'ImportExportPassword='${PASSE} \
                      --form 'ConfigExport=' \
                      ${URL}/cgi-bin/firmwarecfg >TMP1 ; then
    
              echo "Error getting .export for ${HOST} (no data)" >&2
            else
              if [ `stat -c%s TMP1` -lt ${MINSIZE} ] ; then
                echo "Error getting .export for ${HOST} using ${URL} (too small)" >&2
                echo "SID: \"${_SID}\" CHALLENGE: \"${_CHALLENGE}\"  MD5: \"${_MD5}\"  RESPONSE: \"${_RESPONSE}\"" >&2
              else
                chown ${FUSER}:${FGROUP} TMP1
                mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
                LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
                              grep "${IP}.*\.export" | xargs -I{} stat -c '%Y %n' {} | \
                              sort -rn | head -n1 | awk '{print $2}'`
    
                if [ -z "${LATESTCONFIG}" ] ; then                     # No previous .export is found
                  echo "Saving \"${DEST_BASE}.export\" because no previous AVM-config has been found" >&2
                  mv TMP1 "${DEST_BASE}.export"
                else
                  if [ ${TRIED_FREETZ} ] ; then      # The Freetz comparison is better, so make that leading
                    if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
                      echo "Saving \"${DEST_BASE}.export\" because Freetz config has changed" >&2
                      mv TMP1 "${DEST_BASE}.export"
                    fi
                  else
                    if [ `age_of_file "${LATESTCONFIG}"` -gt 30 ] ; then      # Compare EXACTLY if age is older than 30 days
                      if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
                        echo "Saving \"${DEST_BASE}.export\" because latest AVM-config was older than 30 days and is different" >&2
                        mv TMP1 "${DEST_BASE}.export"
                      fi
                    else
                      # remove BINARY data and timestamps before making a diff
                      sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d' "${LATESTCONFIG}" >TMP2
                      sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d' TMP1              >TMP3
                      if ! diff TMP2 TMP3 >/dev/null 2>&1 ; then                  # Compare the 2 stripped configs
                        echo "Saving \"${DEST_BASE}.export\" because AVM-config has changed" >&2
                        diff TMP2 TMP3
                        mv TMP1 "${DEST_BASE}.export"
                      fi
                    fi
                  fi
                fi
              fi
            fi
          fi
        fi
      fi
      cd "${FOLDER}"
      rm -rf ${TMPDIR2}
    }
    
    patch_files ()
    {
      # remove or patch some of the untarred files as they spoil the check for a changed config
      if cd "$1" >/dev/null 2>&1 ; then
        rm -f stat.cfg chrony.drift multid.leases 2>/dev/null
        rm -f fx_cg fonctrl phonebook voipd_call_stat 2>/dev/null
    
        sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' voip.cfg  # remove datestamp in voip.cfg
        sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' ar7.cfg   # remove datestamp in ar7.cfg
        sed -i 's/ *\/\*AVM\*\///g' ar7.cfg                     # remove the /*AVM*/ tag in ar7.cfg
        sed -i 's/; *$/;/g' ar7.cfg                             # remove trailing spaces after a semicolon
        cd - >/dev/null 2>&1
      fi
    }
    
    get_freetz_config ()
    {
      # Fetch config-file with wget in background
      # Somehow it doesn't listen to the 6 seconds timeout I'm giving it
      wget -q --timeout=6                              \
           --http-user=${USER} --http-password=${PASS} \
           ${URL}/cgi-bin/backup/do_backup.cgi         \
           -O TMP_CONFIG 2>/dev/null &
      # Little wait loop to give wget the time to fetch the file
      n=1
      while sleep 1 ; do
        [ `stat -c%s TMP_CONFIG` -ge ${MINSIZE} ] && break
        [ $n -gt 7 ]                              && break
        let n+=1
      done
    
      if [ `stat -c%s TMP_CONFIG` -gt 300 ] ; then
        # If I received at least some data, then wait some more and it might get everything
        sleep 2
      else
        echo "wget didn't get any data from host \"${HOST}\"" >&2
        exec 2>/dev/null
        kill -9 %1 2>/dev/null
        exec 2>&2
        return 1
      fi
      return 0
    }
    
    keep_config_if_different ()
    {
      LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
                          grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | \
                          sort -rn | head -n1 | awk '{print $2}'`
    
      # If no older config is found
      if [ -z "${LATESTCONFIG}" ] ; then
        echo "Saving \"${DEST_BASE}.tgz\" because no previous Freetz config has been found" >&2
        mv TMP_CONFIG "${DEST_BASE}.tgz"
        get_avm_config
      else
    
        TRIED_FREETZ=1  # A flag for get_avm_config to know if the freetz config has been tried or not
        mkdir f1   # folder to untar the current config
        mkdir f2   # folder to untar the previous config
        if ! tar xzf TMP_CONFIG -C f1 2>/dev/null ; then
          echo "I just downloaded an invalid gzipped tar-file" >&2
          rm -f TMP_CONFIG
        else
          if ! tar xzf ${LATESTCONFIG} -C f2 ; then
            echo "A previously downloaded config turns out to be invalid... very strange... (${LATESTCONFIG})" >&2
          else
            # Remove files that are likely to be changed, but don't contain important config
            patch_files f1/var_flash
            patch_files f2/var_flash
    
            if ! diff f1/var_flash f2/var_flash >/dev/null 2>&1 ; then
              echo "Saving \"${DEST_BASE}.tgz\" because Freetz config changed" >&2
              mv TMP_CONFIG "${DEST_BASE}.tgz"
              # Now get the AVM config too
              get_avm_config
            elif [ `find "${DEST_FOLDER}" -maxdepth 1 -type f -name \*.export | wc -l` -lt 1 ] ; then
              # No previous AVM config exists, so try and fetch one anyhow
              get_avm_config
           fi
         fi
       fi
     fi
    }
    
    get_both_configs ()
    {
      PROTO=http
      echo ${PORT} | grep -q '^4[456][0-9]' && PROTO=https
      URL="${PROTO}://${IP}:${PORT}"
    
      if ! httpspeed ${URL} >/dev/null ; then
        # Freetz is not reacting, maybe it is a plain jane AVM Fritzbox
        # I will not be able to see if a config has changed.......
        echo "No reaction from Freetz.... will try AVM now.."
        get_avm_config
      else
        TMPDIR1=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
    
        if ! cd ${TMPDIR1} ; then
          echo "Error changing folder to ${TMPDIR1}" >&2
        else
          touch TMP_CONFIG
    
          get_freetz_config
    
          if [ `stat -c%s TMP_CONFIG` -lt ${MINSIZE} ] ; then
            echo "Error getting config from ${HOST} on ${IP}" >&2
          else
    
            # I think we're getting successful here, let's create a folder already
            chown ${FUSER}:${FGROUP} TMP_CONFIG
            mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
    
            keep_config_if_different
          fi
        fi
        cd "${FOLDER}"
        rm -rf ${TMPDIR1}
      fi
    }
    
    
    # Main Program starts here
    
    if ! cd "${FOLDER}" ; then
      echo "${FOLDER} does not exist" >&2
      exit 1
    fi
    
    BASENAME="${0//*\/}"                           # Strip Path of script name
    BASENAME="${BASENAME%.*}"                      # Strip extension
    CONFIGSOURCE="${FOLDER}/${BASENAME}.list"      # Construct modemlist from script name
    CONFIGFILE=`mktemp`                            # temp file for sanitized modemlist
    
    if [ -n "$*" ] ; then
      echo "$*" >"${CONFIGFILE}"                     # push the commandline parameters in the configfile
    elif [ -e "${CONFIGSOURCE}" ] ; then
      awk -F# '{print $1}' "${CONFIGSOURCE}" | \
          sed '/^ *$/d' >"${CONFIGFILE}"             # Get all the data before # and remove empty lines
    else
      echo "Configfile: \"${CONFIGSOURCE}\" does not exist" >&2
      exit 1
    fi
    
    if [ ! -s "${CONFIGFILE}" ] ; then
      echo "No workable data in \"${CONFIGSOURCE}\"" >&2
      exit 1
    fi
    
    # Main Loop
    FGROUP=`stat -c%G .`
    FUSER=`stat -c%U .`
    
    while read MODEMLINE ; do
      HOST="`echo "${MODEMLINE}"    | awk -F, '{print $1}'`"
      IP="`echo "${MODEMLINE}"      | awk -F, '{print $2}'`"
      PASS="`echo "${MODEMLINE}"    | awk -F, '{print $3}'`"
      PORT="`echo "${MODEMLINE}"    | awk -F, '{print $4}' | tr -cd '0-9'`"
      USER="`echo "${MODEMLINE}"    | awk -F, '{print $5}'`"
      PASSE="`echo "${MODEMLINE}"   | awk -F, '{print $6}'`"
      PORTAVM="`echo "${MODEMLINE}" | awk -F, '{print $7}' | tr -cd '0-9'`"
    
      HOSTORG="${HOST}"
      TRIED_FREETZ=
    
      if [ -z "${IP}" ] ; then
        HOST=`echo "${HOST}" | tr 'A-Z' 'a-z' | \
              egrep -o '[a-z0-9.-]+\.[a-z0-9]+'`     # Loosely Sanitize domain if no IP is given
      fi
      if [ -z "${HOST}" ] ; then
        echo "${HOSTORG} is not a valid host" >&2
      else
        [ ${HEADLESS} ] || echo ${HOST}
        if [ -z "${IP}" ] ; then
          # If HOST is so clearly an IP-address then use that instead....
          if echo "${HOST}" | egrep -q '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$' ; then
            IP=${HOST}
          else
            IP=`host -t A ${HOST} 2>/dev/null | grep -o 'has address .*' | awk '{print $3}' | head -n1`
          fi
        fi
        if [ -z "${IP}" ] ; then
          echo "Unable to resolve ${HOST}" >&2
        else
          [ -z "${PORT}" ]    &&    PORT=${PORT_DEFAULT}      # port of Freetz interface
          [ -z "${PASS}" ]    &&    PASS=${PASS_DEFAULT}      # internal password
          [ -z "${USER}" ]    &&    USER=${USER_DEFAULT}      # external username
          [ -z "${PASSE}" ]   &&   PASSE=${PASS}              # external password
          [ -z "${PORTAVM}" ] && PORTAVM=${PORTAVM_DEFAULT}   # port of AVM interface (https)
    
          DEST_FOLDER="${FOLDER}/${HOST}"
          DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}"
    
          if [ ${PORT} -eq 0 ] ; then
            get_avm_config     # Fetch only AVM-config when the Freetz port is defined as 0
          else
            get_both_configs
          fi
        fi
      fi
    done<"${CONFIGFILE}"
    rm -f "${CONFIGFILE}"
    
     
  3. vice_pres

    vice_pres Mitglied

    Registriert seit:
    6 Apr. 2008
    Beiträge:
    460
    Zustimmungen:
    4
    Punkte für Erfolge:
    18
    Great work! I'll try the new version in the next days.

    I would like to see your updates here :)

    BTW:
    I would like to see a commandline for the list of modems since i'm using .sh on my shellscripts at the end and so my list would be named freetz_getconfigs.sh.list ;)
     
  4. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    404
    Zustimmungen:
    1
    Punkte für Erfolge:
    18
    #24 frater, 21 Mai 2013
    Zuletzt bearbeitet: 24 Mai 2013
    @vice_pres...

    I did some modifications over the weekend and I'm considering the script stable and full featured now.
    I also added some code to get rid of the suffix (in your case .sh), but I just re-read your post and you wanted it in there...

    A bit strange as it was using a "freetz_getconfigs.sh.list" in the old version of the script if the name of the script was "freetz_getconfigs.sh"

    I now especially added code to get rid of the suffix if one exists and am going to leave it that way as it's much neater...

    I tested the script to get the AVM-config from LAN-side. It works!
    It needs to use http instead of https and it does this by looking at the port. If it starts with an "8" it will use http instead of https.

    I enhanced the method to better determine a changed config. Dynamic data like phonebooks are deliberately being ignored.
    I would get too many configs if I didn't (backups should be done automatically).

    I'm keeping the oldest config, because you will then also have the info when that config has been changed.
    The downside of this approach is that you may not have the latest phonebook data in your config.....

    Maybe I will modify the script (by request) if someone would prefer to keep the newest config (and kill the older similar one)
     
  5. vice_pres

    vice_pres Mitglied

    Registriert seit:
    6 Apr. 2008
    Beiträge:
    460
    Zustimmungen:
    4
    Punkte für Erfolge:
    18
    Just a quick note - after upgrading my Ubuntu to 14.04 the script doesn't work because curl changed it behaviour. I had to add --cipher RC4-SHA to every curl call to get the script working again.
     
  6. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    404
    Zustimmungen:
    1
    Punkte für Erfolge:
    18
    #26 frater, 24 Apr. 2014
    Zuletzt bearbeitet: 24 Apr. 2014
    Thanks vice_pres for this feedback!

    I have modified the code based on your findings.
    I do make a difference between https / http connections (in case the program is run from the inside of the Fritzbox).
    No idea what curl does when I supply these options on a plain http, but it is always better to keep code clean and supply parameters as they are expected... (garbage in => garbage out).

    Can you confirm this code is working at your end?

    ~# cat /usr/local/sbin/freetz_getconfigs
    Code:
    #!/bin/sh
    
    # Define constants
    DATESTAMP=`date +%Y-%m-%d.%H-%M`
    FOLDER=/var/www/vhosts/freetz.yourdomain.com/httpdocs
    
    USER_DEFAULT=admin
    PASS_DEFAULT="defaultpass"
    PORT_DEFAULT=61080
    PORTAVM_DEFAULT=450
    MINSIZE=20000
    
    HTTP_OPTIONS=
    HTTPS_OPTIONS='--cipher RC4-SHA'
    
    HEADLESS=
    tty >/dev/null || HEADLESS=true
    
    while getopts cd: opt
    do
      case $opt in
        c)   FOLDER="`pwd`" ;;
        d)   FOLDER="$OPTARG";;
        ?)   printf "Usage: %s [-c] [-d <folder>] [<HOST>],[<IP>],[PASS_FREETZ],[<PORT_FREETZ>],[<USER>],[PASS_EXTERNAL],[PORT_AVM]\n" $0
        exit 2;;
      esac
    done
    shift `expr ${OPTIND} - 1`
    
    
    # Subroutines
    
    age_of_file ()
    {
      if [ ! -e "$1" ] ; then
        echo 0
      else
        AGE_IN_SECONDS=$((`date +%s` - `date +%s -r "$1"` ))
        echo $((${AGE_IN_SECONDS} / 86400))
      fi
    }
    
    httpspeed ()
    {
      # Returns no value and  errorlevel 1 if site is unreachable or too slow
      # Returns the speed in milliseconds if it's below 2 seconds
      _SPEED=`httping -c1 -t1 ${1} 2>/dev/null | \
             egrep '.+/.+/.+/.+/.+' | tail -n1 | \
             awk -F/ '{print $4}' | awk -F. '{print $1}'`
    
      if [ -z "${_SPEED}" ] ; then
        echo -e "${URL} is unreachable\n" >&2
      elif [ ${_SPEED} -gt 2000 ] ; then
        echo "${URL} is slow (${_SPEED} ms)" >&2
      else
        echo ${_SPEED}
        return 0
      fi
      return 1
    }
    
    
    get_avm_config ()
    {
      [ ${PORTAVM} -eq 0 ] && return
    
      TMPDIR2=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
      if ! cd ${TMPDIR2} 2>/dev/null ; then
        echo "Error changing current directory to ${TMPDIR2}. You should never see this message!" >&2
      else
    
        # It's intended to be called from WAN (https)
        # If PORTAVM begins with 8 it assumes LAN and turns to http
        PROTO=https
        echo ${PORTAVM} | grep -q '^80$' && PROTO=http
    
        OPTIONS="${HTTP_OPTIONS}"
        [ "${PROTO}" = 'https' ] && OPTIONS="${HTTPS_OPTIONS}"
    
        URL="${PROTO}://${IP}:${PORTAVM}"
    
        if httpspeed ${URL} >/dev/null ; then
          # get challenge key from FB
    
          _CHALLENGE=`curl -s -k ${OPTIONS} \
                           --user ${USER}:${PASSE} \
                           "${URL}/login.lua" | \
                           grep 'challenge' | egrep '[0-9a-f]{8}' | \
                           awk -F= '{print $2}' | tail -n1 | egrep -o '[0-9a-f]{8}'`
    
          if [ -z "${_CHALLENGE}" ] ; then
            echo "Error getting challenge for ${HOST} using ${URL}" >&2
          else
            # echo "Received Challenge \"${_CHALLENGE}\"" >&2
            # build md5 from challenge key and password
            _MD5=`echo -n ${_CHALLENGE}"-"${PASSE} | \
                  iconv -f ISO8859-1 -t UTF-16LE   | \
                  md5sum -b | awk '{print substr($0,1,32)}'`
    
            # assemble challenge key and md5
            _RESPONSE=${_CHALLENGE}"-"${_MD5}
    
            # get sid for later use
            _SID=`curl -i -s -k ${OPTIONS} \
                        --user ${USER}:${PASSE} \
                        -d 'response='${_RESPONSE} \
                        -d 'page=' \
                        -d 'username='${USER} \
                        ${URL}/login.lua | \
                        grep "Location:" | awk -F'=' {' print $NF '}`
    
            if ! curl -s -k ${OPTIONS} \
                      --user ${USER}:${PASSE} \
                      --form 'sid='${_SID} \
                      --form 'ImportExportPassword='${PASSE} \
                      --form 'ConfigExport=' \
                      ${URL}/cgi-bin/firmwarecfg >TMP1 ; then
    
              echo "Error getting .export for ${HOST} (no data)" >&2
            else
              if [ `stat -c%s TMP1` -lt ${MINSIZE} ] ; then
                echo "Error getting .export for ${HOST} using ${URL} (too small)" >&2
                echo "SID: \"${_SID}\" CHALLENGE: \"${_CHALLENGE}\"  MD5: \"${_MD5}\"  RESPONSE: \"${_RESPONSE}\"" >&2
              else
                chown ${FUSER}:${FGROUP} TMP1
                mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
                LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
                              grep "${IP}.*\.export" | xargs -I{} stat -c '%Y %n' {} | \
                              sort -rn | head -n1 | awk '{print $2}'`
    
                if [ -z "${LATESTCONFIG}" ] ; then                     # No previous .export is found
                  echo "Saving \"${DEST_BASE}.export\" because no previous AVM-config has been found" >&2
                  mv TMP1 "${DEST_BASE}.export"
                else
                  if [ ${TRIED_FREETZ} ] ; then      # The Freetz comparison is better, so make that leading
                    if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
                      echo -e "Saving \"${DEST_BASE}.export\" because Freetz config has changed\n" >&2
                      mv TMP1 "${DEST_BASE}.export"
                    fi
                  else
                    if [ `age_of_file "${LATESTCONFIG}"` -gt 30 ] ; then      # Compare EXACTLY if age is older than 30 days
                      if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
                        echo "Saving \"${DEST_BASE}.export\" because latest AVM-config was older than 30 days and is not EXACTLY the same" >&2
                        mv TMP1 "${DEST_BASE}.export"
                      fi
                    else
    
                      # remove BINARY data and timestamps before making a diff
                      sed    '/ BINFILE:/,/ END OF FILE /d' "${LATESTCONFIG}" >TMP2
                      sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d'            TMP2
                      sed -i '/END OF EXPORT/d;/update_found/d'                TMP2
                      sed -i '/END OF EXPORT/d;/running_version/d'             TMP2
    
                      sed    '/ BINFILE:/,/ END OF FILE /d'             TMP1  >TMP3
                      sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d'            TMP3
                      sed -i '/END OF EXPORT/d;/update_found/d'                TMP3
                      sed -i '/END OF EXPORT/d;/running_version/d'             TMP3
    
                      if ! diff TMP2 TMP3 >/dev/null 2>&1 ; then                  # Compare the 2 stripped configs
                        echo -e "Saving \"${DEST_BASE}.export\" because AVM-config has changed\n" >&2
                        diff TMP2 TMP3 >&2
                        echo -e '\n---\n\n' >&2
                        mv TMP1 "${DEST_BASE}.export"
                      fi
                    fi
                  fi
                fi
              fi
            fi
          fi
        fi
      fi
      cd "${FOLDER}"
      rm -rf ${TMPDIR2}
    }
    
    patch_files ()
    {
      # remove or patch some of the untarred files as they spoil the check for a changed config
      if cd "$1" >/dev/null 2>&1 ; then
        rm -f stat.cfg chrony.drift multid.leases 2>/dev/null
        rm -f fx_cg fonctrl phonebook voipd_call_stat 2>/dev/null
    
        sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' voip.cfg  # remove datestamp in voip.cfg
        sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' ar7.cfg   # remove datestamp in ar7.cfg
        sed -i 's/ *\/\*AVM\*\///g' ar7.cfg                     # remove the /*AVM*/ tag in ar7.cfg
        sed -i 's/; *$/;/g' ar7.cfg                             # remove trailing spaces after a semicolon
        cd - >/dev/null 2>&1
      fi
    }
    
    get_freetz_config ()
    {
      # Fetch config-file with wget in background
      # Somehow it doesn't listen to the 6 seconds timeout I'm giving it
      wget -q --timeout=6                              \
           --http-user=${USER} --http-password=${PASS} \
           ${URL}/cgi-bin/backup/do_backup.cgi         \
           -O TMP_CONFIG 2>/dev/null &
      # Little wait loop to give wget the time to fetch the file
      n=1
      while sleep 1 ; do
        [ `stat -c%s TMP_CONFIG` -ge ${MINSIZE} ] && break
        [ $n -gt 7 ]                              && break
        let n+=1
      done
    
      if [ `stat -c%s TMP_CONFIG` -gt 300 ] ; then
        # If I received at least some data, then wait some more and it might get everything
        sleep 2
      else
        echo "wget didn't get any data from host \"${HOST}\"" >&2
        exec 2>/dev/null
        kill -9 %1 2>/dev/null
        exec 2>&2
        return 1
      fi
      return 0
    }
    
    keep_config_if_different ()
    {
      LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
                          grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | \
                          sort -rn | head -n1 | awk '{print $2}'`
    
      # If no older config is found
      if [ -z "${LATESTCONFIG}" ] ; then
        echo "Saving \"${DEST_BASE}.tgz\" because no previous Freetz config has been found" >&2
        mv TMP_CONFIG "${DEST_BASE}.tgz"
        get_avm_config
      else
    
        TRIED_FREETZ=1  # A flag for get_avm_config to know if the freetz config has been tried or not
        mkdir f1   # folder to untar the current config
        mkdir f2   # folder to untar the previous config
        if ! tar xzf TMP_CONFIG -C f1 2>/dev/null ; then
          echo "I just downloaded an invalid gzipped tar-file" >&2
          rm -f TMP_CONFIG
        else
          if ! tar xzf ${LATESTCONFIG} -C f2 ; then
            echo "A previously downloaded config turns out to be invalid... very strange... (${LATESTCONFIG})" >&2
          else
            # Remove files that are likely to be changed, but don't contain important config
            patch_files f1/var_flash
            patch_files f2/var_flash
    
            if ! diff f1/var_flash f2/var_flash >/dev/null 2>&1 ; then
              echo -e "Saving \"${DEST_BASE}.tgz\" because Freetz config changed\n" >&2
              mv TMP_CONFIG "${DEST_BASE}.tgz"
              # Now get the AVM config too
              get_avm_config
            elif [ `find "${DEST_FOLDER}" -maxdepth 1 -type f -name \*.export | wc -l` -lt 1 ] ; then
              # No previous AVM config exists, so try and fetch one anyhow
              get_avm_config
           fi
         fi
       fi
     fi
    }
    
    get_both_configs ()
    {
      PROTO=http
      echo ${PORT} | grep -q '^4[456][0-9]' && PROTO=https
      URL="${PROTO}://${IP}:${PORT}"
    
      if ! httpspeed ${URL} >/dev/null ; then
        # Freetz is not reacting, maybe it is a plain jane AVM Fritzbox
        # I will not be able to see if a config has changed.......
        echo "No reaction from Freetz (\"${HOST}\").... will try AVM now.."
        get_avm_config
      else
        TMPDIR1=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
    
        if ! cd ${TMPDIR1} ; then
          echo "Error changing folder to ${TMPDIR1}" >&2
        else
          touch TMP_CONFIG
    
          get_freetz_config
    
          if [ `stat -c%s TMP_CONFIG` -lt ${MINSIZE} ] ; then
            echo "Error getting config from ${HOST} on ${IP}" >&2
          else
    
            # I think we're getting successful here, let's create a folder already
            chown ${FUSER}:${FGROUP} TMP_CONFIG
            mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
    
            keep_config_if_different
          fi
        fi
        cd "${FOLDER}"
        rm -rf ${TMPDIR1}
      fi
    }
    
    
    # Main Program starts here
    
    if ! cd "${FOLDER}" ; then
      echo "${FOLDER} does not exist" >&2
      exit 1
    fi
    
    BASENAME="${0//*\/}"                           # Strip Path of script name
    BASENAME="${BASENAME%.*}"                      # Strip extension
    CONFIGSOURCE="${FOLDER}/${BASENAME}.list"      # Construct modemlist from script name
    CONFIGFILE=`mktemp`                            # temp file for sanitized modemlist
    
    if [ -n "$*" ] ; then
      echo "$*" >"${CONFIGFILE}"                     # push the commandline parameters in the configfile
    elif [ -e "${CONFIGSOURCE}" ] ; then
      sed '/^ *$/d;/^ *#/d' "${CONFIGSOURCE}" | \
      awk -F'[\t ]#' '{print $1}' >"${CONFIGFILE}"             # Get all the data before # and remove empty lines
    else
      echo "Configfile: \"${CONFIGSOURCE}\" does not exist" >&2
      exit 1
    fi
    
    if [ ! -s "${CONFIGFILE}" ] ; then
      echo "No workable data in \"${CONFIGSOURCE}\"" >&2
      exit 1
    fi
    
    # Main Loop
    FGROUP=`stat -c%G .`
    FUSER=`stat -c%U .`
    
    while read MODEMLINE ; do
      HOST="`echo "${MODEMLINE}"    | awk -F, '{print $1}' | awk '{print $1}'`"
      IP="`echo "${MODEMLINE}"      | awk -F, '{print $2}' | awk '{print $1}'`"
      PASS="`echo "${MODEMLINE}"    | awk -F, '{print $3}' | awk '{print $1}'`"
      PORT="`echo "${MODEMLINE}"    | awk -F, '{print $4}' | tr -cd '0-9'`"
      USER="`echo "${MODEMLINE}"    | awk -F, '{print $5}' | awk '{print $1}'`"
      PASSE="`echo "${MODEMLINE}"   | awk -F, '{print $6}' | awk '{print $1}'`"
      PORTAVM="`echo "${MODEMLINE}" | awk -F, '{print $7}' | tr -cd '0-9'`"
    
      HOSTORG="${HOST}"
      TRIED_FREETZ=
    
      if [ -z "${IP}" ] ; then
        HOST=`echo "${HOST}" | tr 'A-Z' 'a-z' | \
              egrep -o '[a-z0-9.-]+\.[a-z0-9]+'`     # Loosely Sanitize domain if no IP is given
      fi
      if [ -z "${HOST}" ] ; then
        echo "${HOSTORG} is not a valid host" >&2
      else
        [ ${HEADLESS} ] || echo ${HOST}
        if [ -z "${IP}" ] ; then
          # If HOST is so clearly an IP-address then use that instead....
          if echo "${HOST}" | egrep -q '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$' ; then
            IP=${HOST}
          else
            IP=`host -t A ${HOST} 2>/dev/null | grep -o 'has address .*' | awk '{print $3}' | head -n1`
          fi
        fi
        if [ -z "${IP}" ] ; then
          echo "Unable to resolve ${HOST}" >&2
        else
          [ -z "${PORT}" ]    &&    PORT=${PORT_DEFAULT}      # port of Freetz interface
          [ -z "${PASS}" ]    &&    PASS=${PASS_DEFAULT}      # internal password
          [ -z "${USER}" ]    &&    USER=${USER_DEFAULT}      # external username
          [ -z "${PASSE}" ]   &&   PASSE=${PASS}              # external password
          [ -z "${PORTAVM}" ] && PORTAVM=${PORTAVM_DEFAULT}   # port of AVM interface (https)
    
          DEST_FOLDER="${FOLDER}/${HOST}"
          DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}"
    
          if [ ${PORT} -eq 0 ] ; then
            get_avm_config     # Fetch only AVM-config when the Freetz port is defined as 0
          else
            get_both_configs
          fi
        fi
      fi
    done<"${CONFIGFILE}"
    rm -f "${CONFIGFILE}"
    
     
  7. vice_pres

    vice_pres Mitglied

    Registriert seit:
    6 Apr. 2008
    Beiträge:
    460
    Zustimmungen:
    4
    Punkte für Erfolge:
    18
    #27 vice_pres, 24 Apr. 2014
    Zuletzt bearbeitet: 24 Apr. 2014
    Hi,

    Code is working, I'm using a version I modified a bit (for example storing hostnames in the filename instead of ip because they are dynamic) etc - but tried your code in a new file and it's working :)

    Edit:
    I changed something to the sed when comparing the configs since the passwords seem to be rehashed everytime something is changed via webinterface - so there would always be a diff...
    sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d'
     
  8. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    404
    Zustimmungen:
    1
    Punkte für Erfolge:
    18
    I will look into that sed later. I did notice some configs get an update when in fact nothing really changed.

    I don't understand your remark about the dynamic host.
    My script fully supports dynamic hosts. Just leave out the IP.
     
  9. vice_pres

    vice_pres Mitglied

    Registriert seit:
    6 Apr. 2008
    Beiträge:
    460
    Zustimmungen:
    4
    Punkte für Erfolge:
    18
    Yeah, but the script stores the files with the ip-adress - and I find it more usefull if the script stores the hostname instead.

    Code:
    test.dyndns.org
    Saving "/new/test.dyndns.org/83.20.222.212.2014-04-27.16-06.export" because no previous AVM-config has been found
    Because you set DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}" and I modified it to be DEST_BASE="${DEST_FOLDER}/${HOST}.${DATESTAMP}" :)
     
  10. vice_pres

    vice_pres Mitglied

    Registriert seit:
    6 Apr. 2008
    Beiträge:
    460
    Zustimmungen:
    4
    Punkte für Erfolge:
    18
    Hi,

    Since 06.30 the '--cipher RC4-SHA' option is not needed anymore - in fact it stops the backup from working.
     
  11. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    404
    Zustimmungen:
    1
    Punkte für Erfolge:
    18
    I will check my script and publish the new one...
    I will also check why I have a "new config" each day. Something must have been added that changes every day.
    I need to add something more to the ignore part..

    Will post something soon
     
  12. vice_pres

    vice_pres Mitglied

    Registriert seit:
    6 Apr. 2008
    Beiträge:
    460
    Zustimmungen:
    4
    Punkte für Erfolge:
    18
    I modified the Script to have the HTTPS_Option as an extra option in the config file - since I have some Boxes that don't get 06.30 anymore (7270 for example) and need the cipher. But since I started on an older version of your script a diff is rather long - plus the modifications I wanted (storing in folder named as the hostname etc...). This version works fine for all boxes I'm backing up - the 06.30 Boxes get backed up almost daily (but not daily - some days there are no diffs. I assume it's the call log that changed and triggers an backup)

    Code:
    1c1
    < #!/bin/bash
    ---
    > #!/bin/sh
    5c5
    < FOLDER=/fritzbox/backup
    ---
    > FOLDER=/var/www/vhosts/freetz.yourdomain.com/httpdocs
    8,9c8,9
    < PASS_DEFAULT=topsecret
    < PORT_DEFAULT=6090
    ---
    > PASS_DEFAULT="defaultpass"
    > PORT_DEFAULT=61080
    11a12,15
    >
    > HTTP_OPTIONS=
    > HTTPS_OPTIONS='--cipher RC4-SHA'
    >
    20c24
    <     ?)   printf "Usage: %s [-c] [-d <folder>] [<HOST>],[<IP>],[PASS_FREETZ],[<PORT_FREETZ>],[<USER>],[PASS_EXTERNAL],[PORT_AVM],[CURL_OPTS]\n" $0
    ---
    >     ?)   printf "Usage: %s [-c] [-d <folder>] [<HOST>],[<IP>],[PASS_FREETZ],[<PORT_FREETZ>],[<USER>],[PASS_EXTERNAL],[PORT_AVM]\n" $0
    48c52
    <     echo "${URL} is unreachable" >&2
    ---
    >     echo -e "${URL} is unreachable\n" >&2
    71c75,79
    <     echo ${PORTAVM} | grep -q '^8' && PROTO=http
    ---
    >     echo ${PORTAVM} | grep -q '^80$' && PROTO=http
    >
    >     OPTIONS="${HTTP_OPTIONS}"
    >     [ "${PROTO}" = 'https' ] && OPTIONS="${HTTPS_OPTIONS}"
    >
    74c82
    <     #if httpspeed ${URL} >/dev/null ; then
    ---
    >     if httpspeed ${URL} >/dev/null ; then
    77c85
    <       _CHALLENGE=`curl ${CURLOPTS} -s -k \
    ---
    >       _CHALLENGE=`curl -s -k ${OPTIONS} \
    96c104
    <         _SID=`curl ${CURLOPTS} -i -s -k \
    ---
    >         _SID=`curl -i -s -k ${OPTIONS} \
    104c112
    <         if ! curl ${CURLOPTS} -s -k \
    ---
    >         if ! curl -s -k ${OPTIONS} \
    118c126
    <             mkdir -p "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
    ---
    >             mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
    120c128
    <                           grep "${HOST}.*\.export" | xargs -I{} stat -c '%Y %n' {} | \
    ---
    >                           grep "${IP}.*\.export" | xargs -I{} stat -c '%Y %n' {} | \
    129c137
    <                   echo "Saving \"${DEST_BASE}.export\" because Freetz config has changed" >&2
    ---
    >                   echo -e "Saving \"${DEST_BASE}.export\" because Freetz config has changed\n" >&2
    135c143
    <                     echo "Saving \"${DEST_BASE}.export\" because latest AVM-config was older than 30 days and is different" >&2
    ---
    >                     echo "Saving \"${DEST_BASE}.export\" because latest AVM-config was older than 30 days and is not EXACTLY the same" >&2
    139,144c147,158
    <                   # remove BINARY data, timestamps and passwords (since OS6.0 passwords are rehashed at every configchange through webif it seems) before making a diff
    <                   #sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d' "${LATESTCONFIG}" >TMP2
    <                   #sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d' TMP1              >TMP3
    <                 # just remove timestamps and passwords (since OS6.0 passwords are rehashed at every configchange through webif it seems)
    <                   sed '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d' "${LATESTCONFIG}" >TMP2
    <                   sed '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d' TMP1              >TMP3
    ---
    >
    >                   # remove BINARY data and timestamps before making a diff
    >                   sed    '/ BINFILE:/,/ END OF FILE /d' "${LATESTCONFIG}" >TMP2
    >                   sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d'            TMP2
    >                   sed -i '/END OF EXPORT/d;/update_found/d'                TMP2
    >                   sed -i '/END OF EXPORT/d;/running_version/d'             TMP2
    >
    >                   sed    '/ BINFILE:/,/ END OF FILE /d'             TMP1  >TMP3
    >                   sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d'            TMP3
    >                   sed -i '/END OF EXPORT/d;/update_found/d'                TMP3
    >                   sed -i '/END OF EXPORT/d;/running_version/d'             TMP3
    >
    146,147c160,162
    <                     echo "Saving \"${DEST_BASE}.export\" because AVM-config has changed" >&2
    <                     #diff TMP2 TMP3
    ---
    >                     echo -e "Saving \"${DEST_BASE}.export\" because AVM-config has changed\n" >&2
    >                     diff TMP2 TMP3 >&2
    >                     echo -e '\n---\n\n' >&2
    156c171
    <     #fi
    ---
    >     fi
    209c224
    <                       grep "${HOST}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | \
    ---
    >                       grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | \
    220,221c235,236
    <     mkdir -p f1   # folder to untar the current config
    <     mkdir -p f2   # folder to untar the previous config
    ---
    >     mkdir f1   # folder to untar the current config
    >     mkdir f2   # folder to untar the previous config
    234c249
    <           echo "Saving \"${DEST_BASE}.tgz\" because Freetz config changed" >&2
    ---
    >           echo -e "Saving \"${DEST_BASE}.tgz\" because Freetz config changed\n" >&2
    256c271
    <     echo "No reaction from Freetz.... will try AVM now.."
    ---
    >     echo "No reaction from Freetz (\"${HOST}\").... will try AVM now.."
    274c289
    <         mkdir -p "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
    ---
    >         mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
    300,301c315,316
    <   awk -F# '{print $1}' "${CONFIGSOURCE}" | \
    <       sed '/^ *$/d' >"${CONFIGFILE}"             # Get all the data before # and remove empty lines
    ---
    >   sed '/^ *$/d;/^ *#/d' "${CONFIGSOURCE}" | \
    >   awk -F'[\t ]#' '{print $1}' >"${CONFIGFILE}"             # Get all the data before # and remove empty lines
    317,319c332,334
    <   HOST="`echo "${MODEMLINE}"    | awk -F, '{print $1}'`"
    <   IP="`echo "${MODEMLINE}"      | awk -F, '{print $2}'`"
    <   PASS="`echo "${MODEMLINE}"    | awk -F, '{print $3}'`"
    ---
    >   HOST="`echo "${MODEMLINE}"    | awk -F, '{print $1}' | awk '{print $1}'`"
    >   IP="`echo "${MODEMLINE}"      | awk -F, '{print $2}' | awk '{print $1}'`"
    >   PASS="`echo "${MODEMLINE}"    | awk -F, '{print $3}' | awk '{print $1}'`"
    321,322c336,337
    <   USER="`echo "${MODEMLINE}"    | awk -F, '{print $5}'`"
    <   PASSE="`echo "${MODEMLINE}"   | awk -F, '{print $6}'`"
    ---
    >   USER="`echo "${MODEMLINE}"    | awk -F, '{print $5}' | awk '{print $1}'`"
    >   PASSE="`echo "${MODEMLINE}"   | awk -F, '{print $6}' | awk '{print $1}'`"
    324d338
    <   CURLOPTS="`echo "${MODEMLINE}"   | awk -F, '{print $8}'`"
    355c369
    <       DEST_BASE="${DEST_FOLDER}/${HOST}.${DATESTAMP}"
    ---
    >       DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}"
    
     
  13. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    404
    Zustimmungen:
    1
    Punkte für Erfolge:
    18
    My script is supposed to ignore the call logs as far as I can remember... If I have time I will take a look.
    I haven't noticed that I wasn't getting any configs of some routers...
    Didn't find the time to check. Recently bought my first 3d printer
     
  14. vice_pres

    vice_pres Mitglied

    Registriert seit:
    6 Apr. 2008
    Beiträge:
    460
    Zustimmungen:
    4
    Punkte für Erfolge:
    18
    #34 vice_pres, 9 Jan. 2016
    Zuletzt bearbeitet: 9 Jan. 2016
    Just as a warning: The script can't backup 06.50 at the moment as it seems - and reboots the target (when running 06.50) after trying to get a backup. I'll try if I can take a look why this is happening on monday using my lab...

    This is working from 06.06 (7270 V2) to 7490 (06.50) on all verisons I can test:

    Code:
          _CHALLENGE=`curl ${CURLOPTS} -s -k \
                           --user ${USER}:${PASSE} \
                           "${URL}/login_sid.lua" | \
                           grep Challenge | sed -e 's/.*<Challenge>//' | sed -e 's/<\/Challenge>.*//'`
    
    Code:
            _SID=`curl ${CURLOPTS} -i -s -k \
                        --user ${USER}:${PASSE} \
                        -d 'response='${_RESPONSE} \
                        -d 'page=' \
                        -d 'username='${USER} \
                        ${URL}/login_sid.lua | \
                        grep SID | sed -e 's/.*<SID>//' | sed -e 's/<\/SID>.*//'`
    
    Since there is no login.lua anymore I changed it to login_sid.lua which is available on all 06 Versions I can test. But my testbox doesn't reboot like my production box when testing...
     
  15. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    404
    Zustimmungen:
    1
    Punkte für Erfolge:
    18
    I am very very late implementing your fixes due to all kind of other stuff...
    Here's the complete script again implementing your fixes.....

    I don't know if it still can download the export of old fritzboxes....
    The box on which I was testing it now works.....

    # cat /usr/local/sbin/freetz_getconfigs
    Code:
    #!/bin/sh
    
    
    # Define constants
    DATESTAMP=`date +%Y-%m-%d.%H-%M`
    FOLDER=/var/www/vhosts/freetz.mr-wolf.nl/httpdocs/courant
    
    
    USER_DEFAULT=admin
    PASS_DEFAULT=q1w2e3r4
    PORT_DEFAULT=61080
    PORTAVM_DEFAULT=450
    MINSIZE=20000
    
    
    HTTP_OPTIONS=''
    HTTPS_OPTIONS=''
    # HTTPS_OPTIONS='--cipher RC4-SHA'
    
    
    HEADLESS=
    tty >/dev/null || HEADLESS=true
    
    
    while getopts cd: opt
    do
      case $opt in
        c)   FOLDER="`pwd`" ;;
        d)   FOLDER="$OPTARG";;
        ?)   printf "Usage: %s [-c] [-d <folder>] [<HOST>],[<IP>],[PASS_FREETZ],[<PORT_FREETZ>],[<USER>],[PASS_EXTERNAL],[PORT_AVM]\n" $0
        exit 2;;
      esac
    done
    shift `expr ${OPTIND} - 1`
    
    
    
    
    # Subroutines
    
    
    age_of_file ()
    {
      if [ ! -e "$1" ] ; then
        echo 0
      else
        AGE_IN_SECONDS=$((`date +%s` - `date +%s -r "$1"` ))
        echo $((${AGE_IN_SECONDS} / 86400))
      fi
    }
    
    
    httpspeed ()
    {
      # Returns no value and  errorlevel 1 if site is unreachable or too slow
      # Returns the speed in milliseconds if it's below 2 seconds
      _SPEED=`httping -c1 -t1 ${1} 2>/dev/null | \
             egrep '.+/.+/.+/.+/.+' | tail -n1 | \
             awk -F/ '{print $4}' | awk -F. '{print $1}'`
    
    
      if [ -z "${_SPEED}" ] ; then
        echo -e "${URL} is unreachable\n" >&2
      elif [ ${_SPEED} -gt 2000 ] ; then
        echo "${URL} is slow (${_SPEED} ms)" >&2
      else
        echo ${_SPEED}
        return 0
      fi
      return 1
    }
    
    
    
    
    get_avm_config ()
    {
      [ ${PORTAVM} -eq 0 ] && return
    
    
      TMPDIR2=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
      if ! cd ${TMPDIR2} 2>/dev/null ; then
        echo "Error changing current directory to ${TMPDIR2}. You should never see this message!" >&2
      else
    
    
        # It's intended to be called from WAN (https)
        # If PORTAVM begins with 8 it assumes LAN and turns to http
        PROTO=https
        echo ${PORTAVM} | grep -q '^80$' && PROTO=http
    
    
        CURLOPTS="${HTTP_OPTIONS}"
        [ "${PROTO}" = 'https' ] && CURLOPTS="${HTTPS_OPTIONS}"
    
    
        URL="${PROTO}://${IP}:${PORTAVM}"
    
    
        if httpspeed ${URL} >/dev/null ; then
          # get challenge key from FB
    
    
          _CHALLENGE=`curl ${CURLOPTS} -s -k \
                           --user ${USER}:${PASSE} \
                           "${URL}/login_sid.lua" | \
                           grep 'Challenge' | sed -e 's/.*<Challenge>//' | sed -e 's/<\/Challenge>.*//'`
    
    
          if [ -z "${_CHALLENGE}" ] ; then
            echo "Error getting challenge for ${HOST} using ${URL}/login_sid.lua" >&2
          else
            # echo "Received Challenge \"${_CHALLENGE}\"" >&2
            # build md5 from challenge key and password
            _MD5=`echo -n ${_CHALLENGE}"-"${PASSE} | \
                  iconv -f ISO8859-1 -t UTF-16LE   | \
                  md5sum -b | awk '{print substr($0,1,32)}'`
    
    
            # assemble challenge key and md5
            _RESPONSE=${_CHALLENGE}"-"${_MD5}
    
    
            # get sid for later use
            _SID=`curl ${CURLOPTS} -i -s -k \
                        --user ${USER}:${PASSE} \
                        -d 'response='${_RESPONSE} \
                        -d 'page=' \
                        -d 'username='${USER} \
                        ${URL}/login_sid.lua | \
                        grep SID | sed -e 's/.*<SID>//' | sed -e 's/<\/SID>.*//'`
    
    
            if ! curl ${CURLOPTS} -s -k \
                      --user ${USER}:${PASSE} \
                      --form 'sid='${_SID} \
                      --form 'ImportExportPassword='${PASSE} \
                      --form 'ConfigExport=' \
                      ${URL}/cgi-bin/firmwarecfg >TMP1 ; then
    
    
              echo "Error getting .export for ${HOST} (no data)" >&2
            else
              if [ `stat -c%s TMP1` -lt ${MINSIZE} ] ; then
                echo "Error getting .export for ${HOST} using ${URL} (too small)" >&2
                echo "SID: \"${_SID}\" CHALLENGE: \"${_CHALLENGE}\"  MD5: \"${_MD5}\"  RESPONSE: \"${_RESPONSE}\"" >&2
              else
                chown ${FUSER}:${FGROUP} TMP1
                mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
                LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
                              grep "${IP}.*\.export" | xargs -I{} stat -c '%Y %n' {} | \
                              sort -rn | head -n1 | awk '{print $2}'`
    
    
                if [ -z "${LATESTCONFIG}" ] ; then                     # No previous .export is found
                  echo "Saving \"${DEST_BASE}.export\" because no previous AVM-config has been found" >&2
                  mv TMP1 "${DEST_BASE}.export"
                else
                  if [ ${TRIED_FREETZ} ] ; then      # The Freetz comparison is better, so make that leading
                    if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
                      echo -e "Saving \"${DEST_BASE}.export\" because Freetz config has changed\n" >&2
                      mv TMP1 "${DEST_BASE}.export"
                    fi
                  else
                    if [ `age_of_file "${LATESTCONFIG}"` -gt 30 ] ; then      # Compare EXACTLY if age is older than 30 days
                      if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
                        echo "Saving \"${DEST_BASE}.export\" because latest AVM-config was older than 30 days and is not EXACTLY the same" >&2
                        mv TMP1 "${DEST_BASE}.export"
                      fi
                    else
    
    
                      # remove BINARY data and timestamps before making a diff
                      sed    '/ BINFILE:/,/ END OF FILE /d' "${LATESTCONFIG}" >TMP2
                      sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d'            TMP2
                      sed -i '/END OF EXPORT/d;/update_found/d'                TMP2
                      sed -i '/\$\$\$\$/d;/running_version/d'                  TMP2
    
    
                      sed    '/ BINFILE:/,/ END OF FILE /d'             TMP1  >TMP3
                      sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d'            TMP3
                      sed -i '/END OF EXPORT/d;/update_found/d'                TMP3
                      sed -i '/\$\$\$\$/d;/running_version/d'                  TMP3
    
    
                      if ! diff TMP2 TMP3 >/dev/null 2>&1 ; then                  # Compare the 2 stripped configs
                        echo -e "Saving \"${DEST_BASE}.export\" because AVM-config has changed\n" >&2
                        diff TMP2 TMP3 >&2
                        echo -e '\n---\n\n' >&2
                        mv TMP1 "${DEST_BASE}.export"
                      fi
                    fi
                  fi
                fi
              fi
            fi
          fi
        fi
      fi
      cd "${FOLDER}"
      rm -rf ${TMPDIR2}
    }
    
    
    patch_files ()
    {
      # remove or patch some of the untarred files as they spoil the check for a changed config
      if cd "$1" >/dev/null 2>&1 ; then
        rm -f userstat.cfg stat.cfg chrony.drift multid.leases 2>/dev/null
        rm -f fx_cg fonctrl phonebook voipd_call_stat 2>/dev/null
    
    
        sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' voip.cfg  # remove datestamp in voip.cfg
        sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' ar7.cfg   # remove datestamp in ar7.cfg
        sed -i 's/ *\/\*AVM\*\///g' ar7.cfg                     # remove the /*AVM*/ tag in ar7.cfg
        sed -i 's/; *$/;/g' ar7.cfg                             # remove trailing spaces after a semicolon
        cd - >/dev/null 2>&1
      fi
    }
    
    
    get_freetz_config ()
    {
      # Fetch config-file with wget in background
      # Somehow it doesn't listen to the 6 seconds timeout I'm giving it
      wget -q --timeout=6                              \
           --http-user=${USER} --http-password=${PASS} \
           ${URL}/cgi-bin/backup/do_backup.cgi         \
           -O TMP_CONFIG 2>/dev/null &
      # Little wait loop to give wget the time to fetch the file
      n=1
      while sleep 1 ; do
        [ `stat -c%s TMP_CONFIG` -ge ${MINSIZE} ] && break
        [ $n -gt 7 ]                              && break
        let n+=1
      done
    
    
      if [ `stat -c%s TMP_CONFIG` -gt 300 ] ; then
        # If I received at least some data, then wait some more and it might get everything
        sleep 2
      else
        echo "wget didn't get any data from host \"${HOST}\"" >&2
        exec 2>/dev/null
        kill -9 %1 2>/dev/null
        exec 2>&2
        return 1
      fi
      return 0
    }
    
    
    keep_config_if_different ()
    {
      LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
                          grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | \
                          sort -rn | head -n1 | awk '{print $2}'`
    
    
      # If no older config is found
      if [ -z "${LATESTCONFIG}" ] ; then
        echo "Saving \"${DEST_BASE}.tgz\" because no previous Freetz config has been found" >&2
        mv TMP_CONFIG "${DEST_BASE}.tgz"
        get_avm_config
      else
    
    
        TRIED_FREETZ=1  # A flag for get_avm_config to know if the freetz config has been tried or not
        mkdir f1   # folder to untar the current config
        mkdir f2   # folder to untar the previous config
        if ! tar xzf TMP_CONFIG -C f1 2>/dev/null ; then
          echo "I just downloaded an invalid gzipped tar-file" >&2
          rm -f TMP_CONFIG
        else
          if ! tar xzf ${LATESTCONFIG} -C f2 ; then
            echo "A previously downloaded config turns out to be invalid... very strange... (${LATESTCONFIG})" >&2
          else
            # Remove files that are likely to be changed, but don't contain important config
            patch_files f1/var_flash
            patch_files f2/var_flash
    
    
            if ! diff f1/var_flash f2/var_flash >/dev/null 2>&1 ; then
              [ ${HEADLESS} ] || diff f1/var_flash f2/var_flash >&2
              echo -e "Saving \"${DEST_BASE}.tgz\" because Freetz config changed\n" >&2
              mv TMP_CONFIG "${DEST_BASE}.tgz"
              # Now get the AVM config too
              get_avm_config
            elif [ `find "${DEST_FOLDER}" -maxdepth 1 -type f -name \*.export | wc -l` -lt 1 ] ; then
              # No previous AVM config exists, so try and fetch one anyhow
              get_avm_config
           fi
         fi
       fi
     fi
    }
    
    
    get_both_configs ()
    {
      PROTO=http
      echo ${PORT} | grep -q '^4[456][0-9]' && PROTO=https
      URL="${PROTO}://${IP}:${PORT}"
    
    
      if ! httpspeed ${URL} >/dev/null ; then
        # Freetz is not reacting, maybe it is a plain jane AVM Fritzbox
        # I will not be able to see if a config has changed.......
        echo "No reaction from Freetz (\"${HOST}\").... will try AVM now.."
        get_avm_config
      else
        TMPDIR1=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
    
    
        if ! cd ${TMPDIR1} ; then
          echo "Error changing folder to ${TMPDIR1}" >&2
        else
          touch TMP_CONFIG
    
    
          get_freetz_config
    
    
          if [ `stat -c%s TMP_CONFIG` -lt ${MINSIZE} ] ; then
            echo "Error getting config from ${HOST} on ${IP}" >&2
          else
    
    
            # I think we're getting successful here, let's create a folder already
            chown ${FUSER}:${FGROUP} TMP_CONFIG
            mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
    
    
            keep_config_if_different
          fi
        fi
        cd "${FOLDER}"
        rm -rf ${TMPDIR1}
      fi
    }
    
    
    
    
    # Main Program starts here
    
    
    if ! cd "${FOLDER}" ; then
      echo "${FOLDER} does not exist" >&2
      exit 1
    fi
    
    
    BASENAME="${0//*\/}"                           # Strip Path of script name
    BASENAME="${BASENAME%.*}"                      # Strip extension
    CONFIGSOURCE="${FOLDER}/${BASENAME}.list"      # Construct modemlist from script name
    CONFIGFILE=`mktemp`                            # temp file for sanitized modemlist
    
    
    if [ -n "$*" ] ; then
      echo "$*" >"${CONFIGFILE}"                     # push the commandline parameters in the configfile
    elif [ -e "${CONFIGSOURCE}" ] ; then
      sed '/^ *$/d;/^ *#/d' "${CONFIGSOURCE}" | \
      awk -F'[\t ]#' '{print $1}' >"${CONFIGFILE}"             # Get all the data before # and remove empty lines
    else
      echo "Configfile: \"${CONFIGSOURCE}\" does not exist" >&2
      exit 1
    fi
    
    
    if [ ! -s "${CONFIGFILE}" ] ; then
      echo "No workable data in \"${CONFIGSOURCE}\"" >&2
      exit 1
    fi
    
    
    # Main Loop
    FGROUP=`stat -c%G .`
    FUSER=`stat -c%U .`
    
    
    while read MODEMLINE ; do
      HOST="`echo "${MODEMLINE}"    | awk -F, '{print $1}' | awk '{print $1}'`"
      IP="`echo "${MODEMLINE}"      | awk -F, '{print $2}' | awk '{print $1}'`"
      PASS="`echo "${MODEMLINE}"    | awk -F, '{print $3}' | awk '{print $1}'`"
      PORT="`echo "${MODEMLINE}"    | awk -F, '{print $4}' | tr -cd '0-9'`"
      USER="`echo "${MODEMLINE}"    | awk -F, '{print $5}' | awk '{print $1}'`"
      PASSE="`echo "${MODEMLINE}"   | awk -F, '{print $6}' | awk '{print $1}'`"
      PORTAVM="`echo "${MODEMLINE}" | awk -F, '{print $7}' | tr -cd '0-9'`"
    
    
      HOSTORG="${HOST}"
      TRIED_FREETZ=
    
    
      if [ -z "${IP}" ] ; then
        HOST=`echo "${HOST}" | tr 'A-Z' 'a-z' | \
              egrep -o '[a-z0-9.-]+\.[a-z0-9]+'`     # Loosely Sanitize domain if no IP is given
      fi
      if [ -z "${HOST}" ] ; then
        echo "${HOSTORG} is not a valid host" >&2
      else
        [ ${HEADLESS} ] || echo ${HOST}
        if [ -z "${IP}" ] ; then
          # If HOST is so clearly an IP-address then use that instead....
          if echo "${HOST}" | egrep -q '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$' ; then
            IP=${HOST}
          else
            IP=`host -t A ${HOST} 2>/dev/null | grep -o 'has address .*' | awk '{print $3}' | head -n1`
          fi
        fi
        if [ -z "${IP}" ] ; then
          echo "Unable to resolve ${HOST}" >&2
        else
          [ -z "${PORT}" ]    &&    PORT=${PORT_DEFAULT}      # port of Freetz interface
          [ -z "${PASS}" ]    &&    PASS=${PASS_DEFAULT}      # internal password
          [ -z "${USER}" ]    &&    USER=${USER_DEFAULT}      # external username
          [ -z "${PASSE}" ]   &&   PASSE=${PASS}              # external password
          [ -z "${PORTAVM}" ] && PORTAVM=${PORTAVM_DEFAULT}   # port of AVM interface (https)
    
    
          DEST_FOLDER="${FOLDER}/${HOST}"
          DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}"
    
    
          if [ ${PORT} -eq 0 ] ; then
            get_avm_config     # Fetch only AVM-config when the Freetz port is defined as 0
          else
            get_both_configs
          fi
        fi
      fi
    done<"${CONFIGFILE}"
    rm -f "${CONFIGFILE}"
    
    
    
     
  16. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    404
    Zustimmungen:
    1
    Punkte für Erfolge:
    18
    #36 frater, 28 Feb. 2018
    Zuletzt bearbeitet: 28 Feb. 2018
    The download of AVM's config does not work with 6.83 anymore...
    Anyone has a quick fix?
    Somehow the challenge fails....

    EDIT...
    The root cause must lie somewhere else.
    The script works fine on another 7490 with 6.83 (a higher version, but I will test some more)

    EDIT
    I found the root cause
    It's that two factor challenge method that's spoiling it (where one needs to press a button).
    I just tried a manual backup and that failed for these same reasons.

    # Edit ar7.cfg and change two_factor_auth_enabled
    nvi /var/flash/ar7.cfg

    two_factor_auth_enabled = no;

    # Stop ctlmgr
    ctlmgr -s

    # Start ctlmgr
    ctlmgr