.titleBar { margin-bottom: 5px!important; }

Automatic backup of config using a "pull method"

Dieses Thema im Forum "Freetz" wurde erstellt von frater, 11 Mai 2013.

  1. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    397
    Zustimmungen:
    0
    Punkte für Erfolge:
    16
    #1 frater, 11 Mai 2013
    Zuletzt bearbeitet: 21 Mai 2013
    There is already a thread about an automated backup using a cronjob on the Freetz-box.
    But I would like to get some pointers on how to write an automated script on a linux server that's pulling all the configs..

    I wrote such a script in the past and it was even able to do a mixed set of routers (netscreen, cisco, siemens and speedtouch)....
    Every brand had its specialized method of obtaining a config.

    I would like to write such a script again, but then for freetz
    I can do the whole thing, but I do need to know what to do to get the actual config (using curl or wget)

    This is how I do it for my pfsense router...
    Code:
      wget -qO/dev/null --keep-session-cookies --save-cookies /tmp/pfsense_cookies.txt  --post-data "login=Login&usernamefld=${USER}&passwordfld=${PASS}" ${WGETOPT} ${PROTO}://${IP}:${PORT}/diag_backup.php
      wget -qO${FNAME} --keep-session-cookies --load-cookies /tmp/pfsense_cookies.txt  --post-data 'Submit=download&donotbackuprrd=yes' ${WGETOPT} ${PROTO}://${IP}:${PORT}/diag_backup.php
    
    I would really like to know how to do something similar on a Freetz router.
    Maybe even using the AVM-interface (or both)....



    The whole cronjob looks like this (most code is about getting rid of the config files that are (almost) the same as the previous one):
    Code:
    #!/bin/sh
    
    DATESTAMP=`date +%Y-%m-%d.%H:%M`
    FNAME=pfsense.${DATESTAMP}.xml
    FOLDER=/var/www/vhosts/domain.com/pfsense
    
    USER=admin
    PASS=secret
    
    PROTO=https             # http or https
    IP=8.8.8.8                 # DNS or IP of webif (remote side)
    PORT=9180                # port of webif (remote side)
    WGETOPT=
    
    # turn off certificate checking
    [ "${PROTO}" = "https" ] && WGETOPT="${WGETOPT} --no-check-certificate"
    
    if cd ${FOLDER} ; then
    
      FGROUP=`stat -c%G .`
      FUSER=`stat -c%U .`
      LASTXML=`ls -1t pfsense*xml 2>/dev/null | head -n1`
    
      wget -qO/dev/null --keep-session-cookies --save-cookies /tmp/pfsense_cookies.txt  --post-data "login=Login&usernamefld=${USER}&passwordfld=${PASS}" ${WGETOPT} ${PROTO}://${IP}:${PORT}/diag_backup.php
      wget -qO${FNAME} --keep-session-cookies --load-cookies /tmp/pfsense_cookies.txt  --post-data 'Submit=download&donotbackuprrd=yes' ${WGETOPT} ${PROTO}://${IP}:${PORT}/diag_backup.php
    
      if [ -s ${FNAME} ] ; then
        chown ${FUSER}:${FGROUP} ${FNAME}
    
        if [ ! -z "${LASTXML}" ] ; then
          if [ ! "${LASTXML}" = "${FNAME}" ] ; then
            echo '<time>[0-9]+</time>'       >/tmp/pfgrep.tmp
            echo 'made unknown change'      >>/tmp/pfgrep.tmp
            echo 'Expired .* user accounts' >>/tmp/pfgrep.tmp
    
            egrep -vf /tmp/pfgrep.tmp ${LASTXML} >${LASTXML}.notime
            egrep -vf /tmp/pfgrep.tmp ${FNAME}   >${FNAME}.notime
    
            if diff ${LASTXML}.notime ${FNAME}.notime >${FNAME}.diff ; then
              rm -f ${FNAME}*
            else
              chown ${FUSER}:${FGROUP} ${FNAME}.diff
            fi
    
            rm /tmp/pfgrep.tmp
            rm -f ${LASTXML}.notime 2>/dev/null
            rm -f   ${FNAME}.notime 2>/dev/null
          fi
        fi
      else
        rm -f ${FNAME}
        exit 1
      fi
    else
      exit 1
    fi
    
     
  2. olistudent

    olistudent IPPF-Urgestein

    Registriert seit:
    19 Okt. 2004
    Beiträge:
    14,756
    Zustimmungen:
    2
    Punkte für Erfolge:
    0
    Beruf:
    Softwareentwickler
    Ort:
    Kaiserslautern
    What about tracing a Freetz backup call with Firebug etc. This should reveal the URL and parameters to call...

    Regards
    Oliver
     
  3. MaxMuster

    MaxMuster IPPF-Promi

    Registriert seit:
    1 Feb. 2005
    Beiträge:
    6,919
    Zustimmungen:
    0
    Punkte für Erfolge:
    36
    #3 MaxMuster, 11 Mai 2013
    Zuletzt bearbeitet: 11 Mai 2013
    Code:
    wget --http-user=admin --http-password=ganzgeheim http://fritz.box:81/cgi-bin/backup/do_backup.cgi -q -O BU_$(date +'%Y%m%d').tgz
    
     
  4. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    397
    Zustimmungen:
    0
    Punkte für Erfolge:
    16
    Thank you!!!

    I will come back to this thread after I made a portable script that can be used by others.....
    One will end up with a folder with all the configs sorted....
     
  5. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    397
    Zustimmungen:
    0
    Punkte für Erfolge:
    16
    I wrote this cronjob which uses a .list file containing the IP-addresses of the modems it should fetch the config from.
    That file is read from the same folder where also the configs are placed.

    It's not working as it should because each time the config is download I will get a different file although the config hasn't changed.
    There's probably a good reason for this.

    I wanted to investigate it, so I did a "tar xvzf config.tgz", but that gives me an error.
    Is this normal?


    Here's the script...
    I will improve it somewhat... especially adding some decent logging

    root@ns5:~# cat /usr/local/sbin/freetz_getconfigs
    Code:
    #!/bin/sh
    
    DATESTAMP=`date +%Y-%m-%d.%H:%M`
    FNAME=pfsense.${DATESTAMP}.xml
    FOLDER=/var/www/vhosts/domain.nl/freetz
    
    USER=admin
    PASS_DEFAULT=secret
    PORT_DEFAULT=6080
    
    if cd ${FOLDER} ; then
    
      FGROUP=`stat -c%G .`
      FUSER=`stat -c%U .`
    
      while read MODEMLINE ; do
        IP="`echo "${MODEMLINE}" | awk -F, '{print $1}'`"
        PASS="`echo "${MODEMLINE}" | awk -F, '{print $2}'`"
        PORT="`echo "${MODEMLINE}" | awk -F, '{print $3}' | tr -cd '0-9'`"
    
        if [ -n "${IP}" ] ; then
          [ -z "${PORT}" ] && PORT=${PORT_DEFAULT}
          [ -z "${PASS}" ] && PASS=${PASS_DEFAULT}
    
          FNAME=${IP}.${DATESTAMP}.tgz
    
          if wget --http-user=admin --http-password=${PASS} http://${IP}:${PORT}/cgi-bin/backup/do_backup.cgi -q -O ${FNAME} 2>/dev/null ; then
             if [ -s ${FNAME} ] ; then
               chown ${FUSER}:${FGROUP} ${FNAME}
    
               mkdir ${IP} 2>/dev/null && chown ${FUSER}:${FGROUP} ${IP}
    
               LATESTCONFIG=`find "${FOLDER}/${IP}" -maxdepth 1 -type f -size +20000 | grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | sort -rn | head -n1 | awk '{print $2}'`
               if [ -z "${LATESTCONFIG}" ] ; then
                 mv ${FNAME} ${IP}/
               else
                 if diff ${FNAME} ${LATESTCONFIG} 2>/dev/null ; then
                   mv ${FNAME} ${IP}/
                 else
                   rm -f ${FNAME}
                 fi
               fi
               cd ${FOLDER}
             fi
          else
            echo "Error getting config from ${IP}" 2>/dev/null
          fi
        fi
      done<${0//*\/}.list
    else
      echo "${FOLDER} does not exist" >&2
      exit 1
    fi
    
     
  6. MaxMuster

    MaxMuster IPPF-Promi

    Registriert seit:
    1 Feb. 2005
    Beiträge:
    6,919
    Zustimmungen:
    0
    Punkte für Erfolge:
    36
    What if you do this command alone, without -q ? For me it fails one time and tries again. The second time I get the desired file. It might be easier/better, but at least it worked for me...
     
  7. vice_pres

    vice_pres Mitglied

    Registriert seit:
    6 Apr. 2008
    Beiträge:
    457
    Zustimmungen:
    4
    Punkte für Erfolge:
    18
    I think it would be even more interesting with backing up the fritzbox config itself (the .export file) too - I do it on some freetz boxes with callmonitor described here (http://www.ip-phone-forum.de/showthread.php?t=257074) - but there are 2 Boxes i would like to get the .export from that don't have freetz for a reason... Maybe including the .export in your script would be interessting for you, too.
     
  8. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    397
    Zustimmungen:
    0
    Punkte für Erfolge:
    16
    #8 frater, 12 Mai 2013
    Zuletzt bearbeitet: 13 Mai 2013
    This script is working.....

    It needs a file "freetz_getconfigs.list" in the folder where you want the configs (/var/www/vhosts/domain.com/freetz)

    Each row in that file is a freetz modem with a comma between each parameter....

    <HOST>,<IP>,<PORT>,<PASSWORD>

    I installed it as nightly cronjob

    UPDATED ON: 13-5-2013 13:44

    root@ns5:# cat `which freetz_getconfigs`
    Code:
    #!/bin/sh
    
    DATESTAMP=`date +%Y-%m-%d.%H-%M`
    FNAME=pfsense.${DATESTAMP}.xml
    FOLDER=/var/www/vhosts/domain.com/freetz
    
    USER=admin
    PASS_DEFAULT=secret
    PORT_DEFAULT=6080
    PROTO=http
    MINSIZE=20000
    
    WGETOPT=
    
    patch_files ()
    {
      # remove or patch some of the untarred files as they spoil the check for a changed config
      if cd "$1" >/dev/null 2>&1 ; then
        rm -f stat.cfg chrony.drift multid.leases 2>/dev/null
        rm -f fx_cg fonctrl phonebook voipd_call_stat 2>/dev/null
    
        sed -i '/[0-9][0-9]:[0-9][0-9]:[0-9][0-9]/d' voip.cfg   # remove datestamp in voip.cfg
        sed -i 's/ *\/\*AVM\*\///g' ar7.cfg                     # remove the /*AVM*/ tag in ar7.cfg
        sed -i 's/; *$/;/g' ar7.cfg                             # remove trailing spaces after a semicolon
        cd - >/dev/null 2>&1
      fi
    }
    
    
    if cd ${FOLDER} ; then
    
      FGROUP=`stat -c%G .`
      FUSER=`stat -c%U .`
    
      while read MODEMLINE ; do
        HOST="`echo "${MODEMLINE}" | awk -F, '{print $1}'`"
        IP="`echo "${MODEMLINE}"   | awk -F, '{print $2}'`"
        PASS="`echo "${MODEMLINE}" | awk -F, '{print $3}'`"
        PORT="`echo "${MODEMLINE}" | awk -F, '{print $4}' | tr -cd '0-9'`"
    
        if [ -n "${HOST}" ] && ! echo "${HOST}" | grep -q '^#' ; then
          if [ -z "${IP}" ] ; then
            IP=`host -t A ${HOST} 2>/dev/null | grep -o 'has address .*' | awk '{print $3}' | head -n1`
          fi
          if [ -z "${IP}" ] ; then
            echo "Unable to resolve ${HOST}" >&2
          else
            [ -z "${PORT}" ] && PORT=${PORT_DEFAULT}
            [ -z "${PASS}" ] && PASS=${PASS_DEFAULT}
    
            FNAME=${IP}.${DATESTAMP}.tgz
            touch "${FNAME}"
            wget -q --timeout=6 --http-user=admin --http-password=${PASS} http://${IP}:${PORT}/cgi-bin/backup/do_backup.cgi -O "${FNAME}" 2>/dev/null &
            n=1
            while sleep 1 ; do
              [ `stat -c%s "${FNAME}"` -ge ${MINSIZE} ] && break
              [ $n -gt 7 ] && break
              let n+=1
            done
            if [ `stat -c%s "${FNAME}"` -gt 0 ] ; then
              # If I received at least some data, then wait some more and it might get everything
              sleep 2
            else
              echo "wget didn't get any data from host \"${HOST}\"" >&2
              exec 2>/dev/null
              kill -9 %1 2>/dev/null
              exec 2>&2
            fi
    
            if [ `stat -c%s "${FNAME}"` -lt ${MINSIZE} ] ; then
              echo "Error getting config from ${HOST} on ${IP}" >&2
            else
              chown ${FUSER}:${FGROUP} ${FNAME}
              mkdir ${HOST} 2>/dev/null && chown ${FUSER}:${FGROUP} ${HOST}
    
              LATESTCONFIG=`find "${FOLDER}/${HOST}" -maxdepth 1 -type f -size +10 | grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | sort -rn | head -n1 | awk '{print $2}'`
              # If no older config is found or they have exactly the same name
              if [ -z "${LATESTCONFIG}" ] || [ "${LATESTCONFIG//*\/}" = ${FNAME} ]; then
                mv ${FNAME} ${HOST}/
              else
                rm -rf folder1 2>/dev/null ; mkdir folder1
                rm -rf folder2 2>/dev/null ; mkdir folder2
                if ! tar xzf ${FNAME} -C folder1 2>/dev/null ; then
                  echo "Invalid gzipped tar-file" >&2
                  rm -f ${FNAME}
                else
                  tar xzf ${LATESTCONFIG} -C folder2
    
                  # Remove files that are likely to be changed, but don't contain important config
                  patch_files folder1/var_flash
                  patch_files folder2/var_flash
    
                  if ! diff folder1/var_flash folder2/var_flash >/dev/null 2>&1 ; then
                    mv ${FNAME} ${HOST}/
                  fi
                fi
                rm -rf folder1 2>/dev/null
                rm -rf folder2 2>/dev/null
              fi
            fi
            cd ${FOLDER}
            rm -f "${FNAME}" 2>/dev/null
          fi
        fi
      done<${0//*\/}.list
    else
      echo "${FOLDER} does not exist" >&2
      exit 1
    fi
    
     
  9. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    397
    Zustimmungen:
    0
    Punkte für Erfolge:
    16
    Yes, I would like to have that .export as well, but the link you gave refers to a script pushing that file...
    I have good reasons to choose for a pull system.

    It's the next step.....
    I can do it in one script.....

    What's the wget?
     
  10. vice_pres

    vice_pres Mitglied

    Registriert seit:
    6 Apr. 2008
    Beiträge:
    457
    Zustimmungen:
    4
    Punkte für Erfolge:
    18
    Yeah I know it's a push - and I can't push from 2 Modems i want to backup so I'm using for a pull myself. But I didn't find a way working without callmonitor - you have to login to the webinterface to get the .export
     
  11. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    397
    Zustimmungen:
    0
    Punkte für Erfolge:
    16
    I updated the script.....

    BTW.. the error was caused by the colon in the variable DATESTAMP which was used as part of the filename.
    I went nuts because fetching the file on the prompt was alright and it wasn't in the script.

    The script now seems to work fine....

    In the near future it will take parameters so it can fetch the config of only one host.
    It will then also be possible to call the script from zabbix....
    That way each "zabbix monitored freetz modem" will automatically have their config back-upped and I can also check if this succeeded.....

    If a new file is added it will also get a flag that the config has changed.....

    It feels so good if you have more grip on things....
     
  12. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    397
    Zustimmungen:
    0
    Punkte für Erfolge:
    16
    #12 frater, 13 Mai 2013
    Zuletzt bearbeitet: 13 Mai 2013
    I now have a script I can work with, but a Freetz config doesn't help me when I want to configure a replacement router.
    If I load the tgz-file, I can't login afterwards and probably all the passwords are hashed against its internal serial number.

    I really need a portable .config file
    The one you can download from the AVM-interface using a password.....

    I just installed/started firebug and fetched a config-file, but I have no idea how to crawl for my .config file
     
  13. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    397
    Zustimmungen:
    0
    Punkte für Erfolge:
    16
    #13 frater, 13 Mai 2013
    Zuletzt bearbeitet: 13 Mai 2013
  14. RalfFriedl

    RalfFriedl IPPF-Urgestein

    Registriert seit:
    22 Apr. 2007
    Beiträge:
    12,343
    Zustimmungen:
    0
    Punkte für Erfolge:
    0
    You should also make the user name configurable.
     
  15. vice_pres

    vice_pres Mitglied

    Registriert seit:
    6 Apr. 2008
    Beiträge:
    457
    Zustimmungen:
    4
    Punkte für Erfolge:
    18
    Besides the "LAN-side" Problem there is another Problem:
    Without password provided for the .config the .export is not portable to another box since it's hashed against the SN or something...

    Edit:
    Da wir doch scheinbar alle Deutsch sprechen - können wir das nicht auch hier? Ich finds soooviel einfacher ;)
     
  16. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    397
    Zustimmungen:
    0
    Punkte für Erfolge:
    16
    Although I can speak German, it's not my native tongue and it's much harder for me to express myself.
    Especially technical stuff....

    Daniel Graf has reacted and published something he says is working.....
    https://home.debian-hell.org/blog/2...vm-fritzbox-7390-per-wgetcurl-script-sichern/

    I don't think it is, but I'm sure he will get it going....

    I will then integrate it in my script and we'll have the Freetz-tgz and an AVM .config which is portable....
     
  17. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    397
    Zustimmungen:
    0
    Punkte für Erfolge:
    16
    The AVM-part is not working....
    I've asked Daniel, the one who wrote the base for it (https://home.debian-hell.org/blog/2...vm-fritzbox-7390-per-wgetcurl-script-sichern/) to have a look at it....

    Maybe you can test it on your router and give feedback....
    Most of my routers fail the challenge and the rest don't get a valid config....

    Code:
    #!/bin/sh
    
    DATESTAMP=`date +%Y-%m-%d.%H-%M`
    FNAME=
    FOLDER=/var/www/vhosts/domain.com/freetz
    
    USER_DEFAULT=admin
    PASS_DEFAULT=secret
    PORT_DEFAULT=6080
    PORTS_DEFAULT=450
    MINSIZE=20000
    
    get_avm_config ()
    {
    
    TMP1=`mktemp`
    URL="https://${IP}:${PORTS}"
    
    # get challenge key from FB
    _CHALLENGE=$(curl -s \
                      -k \
                      "${URL}/login.lua" | \
                      grep "^g_challenge" | \
                      awk -F'"' '{ print $2 }')
    
    if [ -z "${_CHALLENGE}" ] ; then
      echo "Error getting challenge for ${HOST} using ${URL}" >&2
    else
      # echo "Received Challenge \"${_CHALLENGE}\"" >&2
      # build md5 from challenge key and password
      _MD5=$(echo -n \
                ${_CHALLENGE}"-"${_PASSWORD} | \
            iconv -f ISO8859-1 \
                  -t UTF-16LE | \
            md5sum -b | \
            awk '{print substr($0,1,32)}')
    
      # assemble challenge key and md5
      _RESPONSE=${_CHALLENGE}"-"${_MD5}
    
      # get sid for later use
      _SID=$(curl -i \
                -s \
                -k \
                -d 'response='${_RESPONSE} \
                -d 'page=' \
                -d 'username='${USER} \
                ${URL}/login.lua | \
            grep "Location:" | \
            awk -F'=' {' print $NF '})
    
      # get configuration from FB and write to STDOUT
      if ! curl -s \
      -k \
      --form 'sid='${_SID} \
      --form 'ImportExportPassword='${PASS} \
      --form 'ConfigExport=' \
      ${URL}/cgi-bin/firmwarecfg >${TMP1} ; then
        echo "Error getting .config for ${HOST} (no data)" >&2
        rm -f ${TMP1} 2>/dev/null
      else
        if [ `stat -c%s "${TMP1}"` -lt 500 ] ; then
          echo "Error getting .config for ${HOST} using ${URL} (too small)" >&2
          rm -f ${TMP1}
        else
          chown ${FUSER}:${FGROUP} ${TMP1}
          mv ${TMP1} ${HOST}/${IP}.${DATESTAMP}.config
        fi
      fi
    
    fi
    }
    
    patch_files ()
    {
      # remove or patch some of the untarred files as they spoil the check for a changed config
      if cd "$1" >/dev/null 2>&1 ; then
        rm -f stat.cfg chrony.drift multid.leases 2>/dev/null
        rm -f fx_cg fonctrl phonebook voipd_call_stat 2>/dev/null
    
        sed -i '/[0-9][0-9]:[0-9][0-9]:[0-9][0-9]/d' voip.cfg   # remove datestamp in voip.cfg
        sed -i 's/ *\/\*AVM\*\///g' ar7.cfg                     # remove the /*AVM*/ tag in ar7.cfg
        sed -i 's/; *$/;/g' ar7.cfg                             # remove trailing spaces after a semicolon
        cd - >/dev/null 2>&1
      fi
    }
    
    
    if cd ${FOLDER} ; then
    
      FGROUP=`stat -c%G .`
      FUSER=`stat -c%U .`
    
      while read MODEMLINE ; do
        HOST="`echo "${MODEMLINE}"  | awk -F, '{print $1}'`"
        IP="`echo "${MODEMLINE}"    | awk -F, '{print $2}'`"
        PASS="`echo "${MODEMLINE}"  | awk -F, '{print $3}'`"
        PORT="`echo "${MODEMLINE}"  | awk -F, '{print $4}' | tr -cd '0-9'`"
        USER="`echo "${MODEMLINE}"  | awk -F, '{print $5}'`"
        PORTS="`echo "${MODEMLINE}" | awk -F, '{print $6}' | tr -cd '0-9'`"
    
        if [ -n "${HOST}" ] && ! echo "${HOST}" | grep -q '^#' ; then
          # echo $HOST
          if [ -z "${IP}" ] ; then
            IP=`host -t A ${HOST} 2>/dev/null | grep -o 'has address .*' | awk '{print $3}' | head -n1`
          fi
          if [ -z "${IP}" ] ; then
            echo "Unable to resolve ${HOST}" >&2
          else
            [ -z "${PORT}" ]  &&  PORT=${PORT_DEFAULT}
            [ -z "${PASS}" ]  &&  PASS=${PASS_DEFAULT}
            [ -z "${USER}" ]  &&  USER=${USER_DEFAULT}
            [ -z "${PORTS}" ] && PORTS=${PORTS_DEFAULT}
    
            FNAME=${IP}.${DATESTAMP}.tgz
            touch "${FNAME}"
            wget -q --timeout=6 --http-user=admin --http-password=${PASS} http://${IP}:${PORT}/cgi-bin/backup/do_backup.cgi -O "${FNAME}" 2>/dev/null &
            n=1
            while sleep 1 ; do
              [ `stat -c%s "${FNAME}"` -ge ${MINSIZE} ] && break
              [ $n -gt 7 ] && break
              let n+=1
            done
            if [ `stat -c%s "${FNAME}"` -gt 0 ] ; then
              # If I received at least some data, then wait some more and it might get everything
              sleep 2
            else
              echo "wget didn't get any data from host \"${HOST}\"" >&2
              exec 2>/dev/null
              kill -9 %1 2>/dev/null
              exec 2>&2
            fi
    
            if [ `stat -c%s "${FNAME}"` -lt ${MINSIZE} ] ; then
              echo "Error getting config from ${HOST} on ${IP}" >&2
            else
              chown ${FUSER}:${FGROUP} ${FNAME}
              mkdir ${HOST} 2>/dev/null && chown ${FUSER}:${FGROUP} ${HOST}
    
              LATESTCONFIG=`find "${FOLDER}/${HOST}" -maxdepth 1 -type f -size +10 | grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | sort -rn | head -n1 | awk '{print $2}'`
              # If no older config is found or they have exactly the same name
              if [ -z "${LATESTCONFIG}" ] || [ "${LATESTCONFIG//*\/}" = ${FNAME} ]; then
                mv ${FNAME} ${HOST}/
                get_avm_config
              else
                rm -rf folder1 2>/dev/null ; mkdir folder1
                rm -rf folder2 2>/dev/null ; mkdir folder2
                if ! tar xzf ${FNAME} -C folder1 2>/dev/null ; then
                  echo "Invalid gzipped tar-file" >&2
                  rm -f ${FNAME}
                else
                  tar xzf ${LATESTCONFIG} -C folder2
    
                  # Remove files that are likely to be changed, but don't contain important config
                  patch_files folder1/var_flash
                  patch_files folder2/var_flash
    
                  if ! diff folder1/var_flash folder2/var_flash >/dev/null 2>&1 ; then
                    mv ${FNAME} ${HOST}/
                    get_avm_config
                  elif [ ! -f ${HOST}/*.config ] ; then
                    # If no .config exists then fetch it anyhow
                    get_avm_config
                  fi
                fi
                rm -rf folder1 2>/dev/null
                rm -rf folder2 2>/dev/null
              fi
            fi
            cd ${FOLDER}
            rm -f "${FNAME}" 2>/dev/null
          fi
        fi
      done<${0//*\/}.list
    else
      echo "${FOLDER} does not exist" >&2
      exit 1
    fi
    
     
  18. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    397
    Zustimmungen:
    0
    Punkte für Erfolge:
    16
    #18 frater, 15 Mai 2013
    Zuletzt bearbeitet: 16 Mai 2013
    It is working now!
    I'm getting a Freetz config and a portable config from the AVM-interface (with a password)
    Thanks to Daniel who figured out how to get the info from the AVM-interface...
    I needed to tinker a bit as I'm not getting the same response from every router.
    I have more than 20 routers to test and there are 4 models (7270v2, 7270v3, 3270, 7390 and 7360)

    Please give me some feedback...
    Look at the code to see how you should make the list (which should be in the folder where you want the configs to be stored)

    Code:
    #!/bin/sh
    
    DATESTAMP=`date +%Y-%m-%d.%H-%M`
    FNAME=
    FOLDER=/var/www/vhosts/domain.com/freetz
    
    USER_DEFAULT=admin
    PASS_DEFAULT=secret
    PORT_DEFAULT=6080
    PORTS_DEFAULT=450
    MINSIZE=20000
    
    get_avm_config ()
    {
    TMPDIR=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
    if cd ${TMPDIR} ; then
    
      URL="https://${IP}:${PORTS}"
    
      SPEED=`httping -c1 -t1 ${URL} 2>/dev/null | \
             egrep '.+/.+/.+/.+/.+' | tail -n1 | \
             awk -F/ '{print $4}' | awk -F. '{print $1}'`
    
      if [ -z "${SPEED}" ] ; then
        echo "${URL} is unreachable" >&2
      elif [ ${SPEED} -gt 3000 ] ; then
        echo "${URL} is slow (${SPEED} ms)" >&2
      else
      # get challenge key from FB
      _CHALLENGE=`curl -s \
                       -k \
                       --user ${USER}:${PASSE} \
                       "${URL}/login.lua" | \
                       grep 'challenge' | egrep '[0-9a-f]{8}' | \
                       awk -F= '{print $2}' | tail -n1 | \
                       egrep -o '[0-9a-f]{8}'`
    
      if [ -z "${_CHALLENGE}" ] ; then
        echo "Error getting challenge for ${HOST} using ${URL}" >&2
    
      else
        # echo "Received Challenge \"${_CHALLENGE}\"" >&2
        # build md5 from challenge key and password
        _MD5=$(echo -n \
                ${_CHALLENGE}"-"${PASSE} | \
            iconv -f ISO8859-1 \
                  -t UTF-16LE | \
            md5sum -b | \
            awk '{print substr($0,1,32)}')
    
        # assemble challenge key and md5
        _RESPONSE=${_CHALLENGE}"-"${_MD5}
    
        # get sid for later use
        _SID=$(curl -i \
                -s \
                -k \
                --user ${USER}:${PASSE} \
                -d 'response='${_RESPONSE} \
                -d 'page=' \
                -d 'username='${USER} \
                ${URL}/login.lua | \
            grep "Location:" | \
            awk -F'=' {' print $NF '})
    
        # get configuration from FB and write to STDOUT
        if ! curl -s \
          -k \
          --user ${USER}:${PASSE} \
          --form 'sid='${_SID} \
          --form 'ImportExportPassword='${PASSE} \
          --form 'ConfigExport=' \
          ${URL}/cgi-bin/firmwarecfg >TMP1 ; then
          echo "Error getting .config for ${HOST} (no data)" >&2
        else
          if [ `stat -c%s TMP1` -lt ${MINSIZE} ] ; then
            echo "Error getting .config for ${HOST} using ${URL} (too small)" >&2
            echo "SID: \"${_SID}\" CHALLENGE: \"${_CHALLENGE}\"  MD5: \"${_MD5}\"  RESPONSE: \"${_RESPONSE}\"" >&2
          else
            chown ${FUSER}:${FGROUP} TMP1
            mv TMP1 ${FOLDER}/${HOST}/${IP}.${DATESTAMP}.config
          fi
        fi
      fi
      fi
    fi
    cd ${FOLDER}
    rm -rf ${TMPDIR}
    }
    
    patch_files ()
    {
      # remove or patch some of the untarred files as they spoil the check for a changed config
      if cd "$1" >/dev/null 2>&1 ; then
        rm -f stat.cfg chrony.drift multid.leases 2>/dev/null
        rm -f fx_cg fonctrl phonebook voipd_call_stat 2>/dev/null
    
        sed -i '/[0-9][0-9]:[0-9][0-9]:[0-9][0-9]/d' voip.cfg   # remove datestamp in voip.cfg
        sed -i 's/ *\/\*AVM\*\///g' ar7.cfg                     # remove the /*AVM*/ tag in ar7.cfg
        sed -i 's/; *$/;/g' ar7.cfg                             # remove trailing spaces after a semicolon
        cd - >/dev/null 2>&1
      fi
    }
    
    
    if cd ${FOLDER} ; then
    
      FGROUP=`stat -c%G .`
      FUSER=`stat -c%U .`
    
      while read MODEMLINE ; do
        HOST="`echo "${MODEMLINE}"  | awk -F, '{print $1}'`"
        IP="`echo "${MODEMLINE}"    | awk -F, '{print $2}'`"
        PASS="`echo "${MODEMLINE}"  | awk -F, '{print $3}'`"
        PORT="`echo "${MODEMLINE}"  | awk -F, '{print $4}' | tr -cd '0-9'`"
        USER="`echo "${MODEMLINE}"  | awk -F, '{print $5}'`"
        PASSE="`echo "${MODEMLINE}" | awk -F, '{print $6}'`"
        PORTS="`echo "${MODEMLINE}" | awk -F, '{print $7}' | tr -cd '0-9'`"
    
        if [ -n "${HOST}" ] && ! echo "${HOST}" | grep -q '^#' ; then
          echo $HOST
          if [ -z "${IP}" ] ; then
            IP=`host -t A ${HOST} 2>/dev/null | grep -o 'has address .*' | awk '{print $3}' | head -n1`
          fi
          if [ -z "${IP}" ] ; then
            echo "Unable to resolve ${HOST}" >&2
          else
            [ -z "${PORT}" ]  &&  PORT=${PORT_DEFAULT}      # port of Freetz interface
            [ -z "${PASS}" ]  &&  PASS=${PASS_DEFAULT}      # internal password
            [ -z "${USER}" ]  &&  USER=${USER_DEFAULT}      # external username
            [ -z "${PASSE}" ] && PASSE=${PASS}              # external password
            [ -z "${PORTS}" ] && PORTS=${PORTS_DEFAULT}     # port of AVM interface (https)
    
            FNAME=${IP}.${DATESTAMP}.tgz
            touch "${FNAME}"
            wget -q --timeout=6 --http-user=admin --http-password=${PASS} http://${IP}:${PORT}/cgi-bin/backup/do_backup.cgi -O "${FNAME}" 2>/dev/null &
            n=1
            while sleep 1 ; do
              [ `stat -c%s "${FNAME}"` -ge ${MINSIZE} ] && break
              [ $n -gt 7 ] && break
              let n+=1
            done
            if [ `stat -c%s "${FNAME}"` -gt 0 ] ; then
              # If I received at least some data, then wait some more and it might get everything
              sleep 2
            else
              echo "wget didn't get any data from host \"${HOST}\"" >&2
              exec 2>/dev/null
              kill -9 %1 2>/dev/null
              exec 2>&2
            fi
    
            if [ `stat -c%s "${FNAME}"` -lt ${MINSIZE} ] ; then
              echo "Error getting config from ${HOST} on ${IP}" >&2
            else
              chown ${FUSER}:${FGROUP} ${FNAME}
              mkdir ${FOLDER}/${HOST} 2>/dev/null && chown ${FUSER}:${FGROUP} ${FOLDER}/${HOST}
    
              LATESTCONFIG=`find "${FOLDER}/${HOST}" -maxdepth 1 -type f -size +10 | grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | sort -rn | head -n1 | awk '{print $2}'`
              # If no older config is found or they have exactly the same name
              if [ -z "${LATESTCONFIG}" ] || [ "${LATESTCONFIG//*\/}" = ${FNAME} ]; then
                mv ${FNAME} ${FOLDER}/${HOST}/
                get_avm_config
              else
                rm -rf folder1 2>/dev/null ; mkdir folder1
                rm -rf folder2 2>/dev/null ; mkdir folder2
                if ! tar xzf ${FNAME} -C folder1 2>/dev/null ; then
                  echo "Invalid gzipped tar-file" >&2
                  rm -f ${FNAME}
                else
                  tar xzf ${LATESTCONFIG} -C folder2
    
                  # Remove files that are likely to be changed, but don't contain important config
                  patch_files folder1/var_flash
                  patch_files folder2/var_flash
    
                  if ! diff folder1/var_flash folder2/var_flash >/dev/null 2>&1 ; then
                    mv ${FNAME} ${FOLDER}/${HOST}/
                    get_avm_config
                  elif [ `find ${FOLDER}/${HOST} -maxdepth 0 -type f -name \*.config | wc -l` -lt 1 ] ; then
                    # If no .config exists then fetch it anyhow
                    get_avm_config
                  fi
                fi
                rm -rf folder1 2>/dev/null
                rm -rf folder2 2>/dev/null
              fi
            fi
            cd ${FOLDER}
            rm -f "${FNAME}" 2>/dev/null
          fi
        fi
      done<${0//*\/}.list
    else
      echo "${FOLDER} does not exist" >&2
      exit 1
    fi
    
     
  19. frater

    frater Mitglied

    Registriert seit:
    23 Nov. 2008
    Beiträge:
    397
    Zustimmungen:
    0
    Punkte für Erfolge:
    16
    I just updated the script to correct a little error (checking if an AVM config has been made at all for a specific host)

    I will clean this script up in the near future...
    It's a bit of a mess with too many nests in 1 procedure....

    This is only to make the script better readable and therefore better to debug.

    You need to create a list with all the modems you want to check.
    The filename of the list is the same as the name of the script with ".list" attached.

    If this list only has a host-address it will use the default values for the Freetz-interface and AVM-interface.
    The parameters are separated with comma's

    <HOST>,<IP>,<PASSfreetz>,<PORTfreetz>,<USER>,<PASSavm>,<PORTavm>

    If the first HOST is resolvable to an IP you don't need to give the IP.
    The optional <IP> overrides <HOST>
    <HOST> will then only be used as the name of the folder in which the configs will come.
     
  20. vice_pres

    vice_pres Mitglied

    Registriert seit:
    6 Apr. 2008
    Beiträge:
    457
    Zustimmungen:
    4
    Punkte für Erfolge:
    18
    Your script is working for 2 7270 that I use to pull the config from, didn't test the others since they are pulling. I modified it a bit since the 2 Boxes I need don't have freetz and since the script don't need to try to get the .tgz. The Script from Daniel didn't work for me in the first place so I took your script and ripped the freetz parts out of it :)

    I also inserted a small
    Code:
            mkdir -p ${FOLDER}/${HOST}
            chown ${FUSER}:${FGROUP} ${FOLDER}/${HOST}
    
    before moving the .config file since I don't want to create the folder by myself if I add another host

    Thanks alot!