Script for wget

hrust_ray

Neuer User
Mitglied seit
11 Jan 2007
Beiträge
133
Punkte für Reaktionen
0
Punkte
0
Hallo.
I want to make downloads with wget easier.
Idea is following. All links for downloads are placed in one file.
This file is parsed using grep and for each link wget is started.
If wget stopped with error, it should be started again with oprion -c (retry) and so on - untill file is downloaded completely.
I'l to write such a script, have 2 questions.
1. how can I detect result of wget execution? if it was timeout, error or success?
2. About controlling of wget processes. It will be nice if wget can save percent of downloaded data to file, that can be parsed from outside. If I redirect output or wget to file (wget ... > log.txt), it will write data each second and finally file size will be huge?
Or its better when each wget process logs to its own file and then all this files can be parsed using grep. What do U think?
 
I'd like do it without dsmod, only scripts
 
Hallo, me again.
I need help in script, I created. I have 2 scripts:
wget.sh - script take in argument link, downloads file with retries and save log info for result.
wget_list.sh - take in argument text file with links and for each link calls wget.sh.
Here are scripts:

wget_list.sh
Code:
#!/bin/sh
echo "start"
REPORT="/var/media/ftp/ROOT/Bin/wget/ready.txt"

for str1 in $(cat $1 | grep "/"); do
	echo "line _ $str1 _"
	./wget.sh "$str1"
	echo "$str1"
done
echo "LIST finished:`date +%D_%H:%M` : $1">> $REPORT

wget.sh
Code:
#!/bin/sh
REPORT="/var/media/ftp/ROOT/Bin/wget/ready.txt"
RETRIES1=10
RETRIES2=10
result=1
nRetries=$RETRIES
echo "starting:`date +%D_%H:%M` : $1">> $REPORT 
while [ "$result" != "0" ]; do
	wget -c $1
	result=$?
	sleep 5
	let "nRetries -= 1"
	if [ "$nRetries" = "0" ]; then
		nRetries=$RETRIES2
		while [ "$result" != "0" ]; do
			wget $1
			result=$?
			sleep 20
			let "nRetries -= 1"
			if [ "$nRetries" = "0" ]; then
				echo "__failed!_:`date +%D_%H:%M` : $1">> $REPORT 
				exit
			fi
		done
		echo "__failed!_:`date +%D_%H:%M` : $1">> $REPORT 
		exit
	fi
done
echo "finished:`date +%D_%H:%M` : $1">> $REPORT

download.txt:
Code:
http://download.microsoft.com/download/A/9/1/A91D6B2B-A798-47DF-9C7E-A97854B7DD18/VC.iso
http://download.microsoft.com/download/8/1/d/81d3f35e-fa03-485b-953b-ff952e402520/VS2008ProEdition90dayTrialENUX1435622.iso
http://download.microsoft.com/download/c/7/c/c7c41934-8016-44c2-807c-d0301899f5eb/VS2008MSDNLibraryENUX1429217.iso
http://download.microsoft.com/download/d/2/e/d2e43e33-53b0-45b6-ba70-fe6fdc4aa5bb/Windows%20Mobile%205.0%20Pocket%20PC%20SDK.msi
http://download.microsoft.com/download/1/1/1/1116b75a-9ec3-481a-a3c8-1777b5381140/vcredist_x86.exe
http://download.microsoft.com/download/c/7/d/c7d9b927-f4e6-4ab2-8399-79a2d5cdfac9/VS80sp1-KB932232-X86-ENU.exe
http://download.microsoft.com/download/9/9/5/995f050f-3e12-4a37-9b38-bec3b3e899da/msdnixp.exe
http://download.microsoft.com/download/e/a/4/ea4b699b-bec4-4722-96d3-254580ed7f9e/SQLEXPR32.EXE

if I test wget.sh with first line from download.txt as a parameter - it works.
But if I start "./wget_list.sh ./download.txt", I get very strange result:

Code:
# ./wget_list.sh ./download.txt
start
 _ne _ http://download.microsoft.com/download/A/9/1/A91D6B2B-A798-47DF-9C7E-A97854B7DD18/VC.iso
: Invalid argument

It looks like some control symbols are interpreted. Very strange, why it works when I give this command directly in terminal!?
 
Zuletzt bearbeitet:
You probably wrote your file download.txt on Windows. Change the line ends from CRLF (Windows) to LF (Unix). You can use dos2unix or something like that.

Or you can change your skript to delete the CR:
Code:
...
for str1 in $(cat $1 | grep "/" [B]| tr -d '\r'[/B]); do
...
 

Zurzeit aktive Besucher

Neueste Beiträge

Statistik des Forums

Themen
244,640
Beiträge
2,215,722
Mitglieder
371,219
Neuestes Mitglied
csgaming
Holen Sie sich 3CX - völlig kostenlos!
Verbinden Sie Ihr Team und Ihre Kunden Telefonie Livechat Videokonferenzen

Gehostet oder selbst-verwaltet. Für bis zu 10 Nutzer dauerhaft kostenlos. Keine Kreditkartendetails erforderlich. Ohne Risiko testen.

3CX
Für diese E-Mail-Adresse besteht bereits ein 3CX-Konto. Sie werden zum Kundenportal weitergeleitet, wo Sie sich anmelden oder Ihr Passwort zurücksetzen können, falls Sie dieses vergessen haben.