Article 3893 of comp.lang.perl: Xref: feenix.metronet.com comp.lang.perl:3893 Path: feenix.metronet.com!news.ecn.bgu.edu!mp.cs.niu.edu!ux1.cso.uiuc.edu!howland.reston.ans.net!gatech!asuvax!ncar!vexcel!copper!aspen.craycos.com!scott From: scott@craycos.com (Scott Bolte) Newsgroups: comp.lang.perl Subject: Re: ftpget script (provided) Message-ID: <1993Jun30.175050.14781@craycos.com> Date: 30 Jun 93 17:50:50 GMT References: Organization: Cray Computer Corporation Lines: 832 In article mathias@solomon.technet.sg (Mathias Koerber) writes: ... >What I'd like is a script that just goes to a specified host, logs in, >cd's to the directory and then ftps the named files. Would be nice if it >sat atop ftp.pl. > >Does anyone already have such a beast? I wrote just such a beast a while ago and have been using it ever since. (In fact, it was used to pick up perl 4.036 just two nights ago.) It does not use ftp.pl, all it requires is ftp. If you have the at command available it will even do the transfers at night to ease the load on the net. Here is a brief description of the scripts shar'ed together below: ftpget Obtains a single file from a remote host. ftpmget Obtains a set of files from a remote host. ftpls Run ls -l in a specific directory on a remote host. ftplist Given a set of hosts & directories make a series of calls to ftpls. The results are placed in files that reflect the listing source. Enjoy, Scott ___________________________________________________________________________ Scott Bolte scott@craycos.com +1 719 540 4186 Cray Computer Corporation, 1110 Bayfield Drive, Colorado Springs, CO 80906 As anyone here will tell you: I speak for myself. *** On the Internet no one can hear you scream *** -------- rip, cut, sever, slice, dice or tear here -------------------------- #! /bin/sh # This is a shell archive. Remove anything before this line, then unpack # it by saving it into a file and typing "sh file". To overwrite existing # files, type "sh file -c". You can also feed this as standard input via # unshar, or by typing "sh 'README' <<'END_OF_FILE' X Ftp Perl Scripts, Version 0.1 X X Copyright (c) 1993, Scott Bolte X These scripts are free software. But if you want to send money anyway I won't mind. The scripts are distributed in the hope that they will be useful, but WITHOUT ANY WARRANTY. X You can take the scripts and use them yourself. You can even include them in a product you sell as long as they do not add to the product cost. But the copyright must be maintained. X The files in this directory were cobbled together in the last few weeks. I have been using to automate ftp transfers. They also can be asked to run at night in order to spread the demands on the net. X X ftpget Obtains a single file from a remote host. X X ftpmget Obtains a set of files from a remote host. X X ftpls Run ls -l in a specific directory on a remote X host. X X ftplist Given a set of hosts & directories make a X series of calls to ftpls. The results are X placed in files that reflect the listing X source. X X ftpvaporware A script, not yet written, that can look at the X data provided by ftplist. It will provide X notification that a item on an archive has X changed. Or that a new release of a program is X now available. X I would appreciate if any additions you make are sent to me. I will try to incorporate them in any future releases. X X Scott Bolte (scott@craycos.com) X 1993.06.30 END_OF_FILE if test 1371 -ne `wc -c <'README'`; then echo shar: \"'README'\" unpacked with wrong size! fi # end of 'README' fi if test -f 'ftpget.pl' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'ftpget.pl'\" else echo shar: Extracting \"'ftpget.pl'\" \(5666 characters\) sed "s/^X//" >'ftpget.pl' <<'END_OF_FILE' X#!/bin/perl X X############################################################################# X# X# Copyright 1993 Scott Bolte (scott@craycos.com) X# X# Leave this copyright alone. But feel free to do with the script X# as you please. Sending me enhancements would be appreciated. X# X# If you feel like pretending this is shareware, and want to X# send some money my way, feel free. I promise not to object. X# X# Summary of ftpget: X# X# Obtain a file via ftp from a remote system. Anonymous ftp X# is used. Can be asked to delay the actual ftp request. X# On failure print out a call that can be used to try again. X# X# History: X# X# 1993.06.14 Initial implementation. X# X# 1993.06.15 Made remote and local file name processing safe X# even when the file name contains white space. X# X# 1993.06.28 Added interface to use "at" internally. X# X# Examples: X# X# ftpget --at 23:30 prep.ai.mit.edu pub/gnu/perl-4.036.tar.gz X# X# ftpget --at 23:30 prep.ai.mit.edu pub/gnu/perl-4.036.tar.gz new-perl X# X# If you have an ftp hierarchy, as I do, the remote system can be X# derived from the current path. X# X# cd ~/ftp/prep.ai.mit.edu X# ftpget --at 23:30 - pub/gnu/perl-4.036.tar.gz X# X############################################################################# X X$zero = $0; X$zero =~ s,.*/,,; X sub usage { X print < 2 ) { X &usage(); X exit(1); X} X$system = $ARGV[0]; X$remote_file = $ARGV[1]; X X# X# If the system spec was "-" try to figure out where we are. From that X# we might be able to construct a default host. X# if ( $system eq "-" ) { X $system = `/bin/pwd`; X $system =~ s/\n//; X $original = $system; X $system =~ s,^.*/ftp/,,; X $system =~ s,/.*$,,; X die("Could not determine system given path \"$original\".\n") X if $system eq ""; X print(STDERR "Derived system is \"$system\".\n") if $verbose; X} X X# X# Either take the optional third argument or construct it from the second. X# X if ( "$remote_file" eq "" ) { X print(STDERR "Must specify a non-null file name.\n"); X exit(1); X} if ( $#ARGV == 2 ) { X $local_file = $ARGV[2]; X $explicit = 1; X} else { X $local_file = $remote_file; X $local_file =~ s,/+$,,; X $local_file =~ s,.*/,,; X $explicit = 0; X X print(STDERR "Derived local path is \"$local_file\".\n") if $verbose; X if ( "$local_file" eq "" ) { X print(STDERR "Unable to construct a local filename.\n"); X exit(1); X } X} X X X# X# Make the path specifications safe even when they contain spaces. X# X$safe_remote = $remote_file; X$safe_local = $local_file; X$safe_remote =~ s/(.*)/"$1"/ if $safe_remote =~ /\s/; X$safe_local =~ s/(.*)/"$1"/ if $safe_local =~ /\s/; X X# X# If a delay was asked for run the command later. X# if ( $delay_time ne "" ) { X $me = $0; X $me .= " --verbose" if $verbose; X $me .= " --debug" if $debug; X $me .= " $system $safe_remote"; X $me .= " $safe_local" if $explicit; X $at = "at $delay_time"; X print(STDERR "At $delay_time the following command will be run:\n"); X print(STDERR " $me\n"); X if ( $debug ) { X print(STDERR "Skipping command in debug mode.\n"); X exit(0); X } X open(CMD, "|$at") || die("Could not run command ($at). $!\n"); X print(CMD $me); X close(CMD); X exit(0); X} X X# X# Build the batch ftp command. X# X$user = (getpwuid($<))[0]; # safe when run from "at". X$localhost = `hostname`; chop($localhost); if ( $localhost !~ /\./ ) { X # X # If the host name does not have '.' notation try to X # get an alias. We then hope it is in domain name notation. X # X @fullhost = gethostbyname($localhost); X $localhost = $fullhost[1] if $fullhost[1] ne ""; X} X$template = X" open $system X user anonymous ${zero}4$user@$localhost X bin X get $safe_remote $safe_local X bye X"; X X# X# Run the batch ftp command. X# print(STDERR "Running command ($cmd).\n") if $verbose; open(CMD, "|$cmd") || die("Could not start command ($cmd). $!\n"); print CMD $template; close(CMD); X X X# Note whether or not the local file was obtained. X# if ( -f $local_file ) { X print("Obtained \"$local_file\" from $system.\n"); X} else { X $pwd=`pwd`; X chop($pwd); X @time = localtime($time); X $next = sprintf("%2d:%02d", $time[2], $time[1]); X X $sep = "\t\\\n\t "; X $cmd = "$0"; X $cmd .= "$sep--at $next"; X $cmd .= "$sep$system"; X $cmd .= "$sep$safe_remote"; X $cmd .= "$sep$safe_local" if $explicit; X X print(" Unable to obtained \"$local_file\" from $system. To try again the following command might be used: X cd $pwd; X$cmd X"); X} END_OF_FILE if test 5666 -ne `wc -c <'ftpget.pl'`; then echo shar: \"'ftpget.pl'\" unpacked with wrong size! fi chmod +x 'ftpget.pl' # end of 'ftpget.pl' fi if test -f 'ftplist.pl' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'ftplist.pl'\" else echo shar: Extracting \"'ftplist.pl'\" \(1543 characters\) sed "s/^X//" >'ftplist.pl' <<'END_OF_FILE' X#!/bin/perl X X############################################################################# X# X# Copyright 1993 Scott Bolte (scott@craycos.com) X# X# Leave this copyright alone. But feel free to do with the script X# as you please. Sending me enhancements would be appreciated. X# X# If you feel like pretending this is shareware, and want to X# send some money my way, feel free. I promise not to object. X# X# Summary of ftplist: X# X# Run ftpls on a bunch of systems. The results are put in files X# whose names map to the system/directory pair. Older copies are X# renamed before the new edition is obtained. X# X# The expectation is that additional scripts will be run after X# this one. They will compare the old and new listings to note X# changes. X# X# History: X# X# 1993.06.28 Initial implementation. X# X# Examples: X# X# ftplist X# X############################################################################# X X%set = ( X "agate.berkeley.edu", "pub/386BSD/386bsd-0.1/unofficial", X "bsd.coe.montana.edu", "pub/patch-kit", X "hrd769.brooks.af.mil", "pub/FAQ", X "prep.ai.mit.edu", "pub/gnu", X ); X foreach $system (sort(keys(%set))) { X $file = "$system:$set{$system}"; X $file =~ s,/,_,g; X $old = "$file.OLD"; X if ( -f $file ) { X $error = "Could not rename \"$file\" to \"$old\". $!\n"; X $error .= "New listing of $set{$system} will not be obtained.\n"; X rename($file, $old) || (warn($error), next); X } X $cmd = "ftpls $system $set{$system} > $file"; X system($cmd) && die("Could not run command ($cmd). $!\n"); X} END_OF_FILE if test 1543 -ne `wc -c <'ftplist.pl'`; then echo shar: \"'ftplist.pl'\" unpacked with wrong size! fi # end of 'ftplist.pl' fi if test -f 'ftpls.pl' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'ftpls.pl'\" else echo shar: Extracting \"'ftpls.pl'\" \(1853 characters\) sed "s/^X//" >'ftpls.pl' <<'END_OF_FILE' X#!/bin/perl X X############################################################################# X# X# Copyright 1993 Scott Bolte (scott@craycos.com) X# X# Leave this copyright alone. But feel free to do with the script X# as you please. Sending me enhancements would be appreciated. X# X# If you feel like pretending this is shareware, and want to X# send some money my way, feel free. I promise not to object. X# X# Summary of ftpls: X# X# Do a ls in a specific directory on a remote ftp system. X# Anonymous ftp is used. X# X# History: X# X# 1993.06.28 Initial implementation. X# X# Examples: X# X# ftpls remote_system remote_dir X# X############################################################################# X X$cmd = "ftp -n"; # change "ftp -n" to "cat -n" for testing. X$zero = $0; X$zero =~ s,.*/,,; X X############################################################################# X# X# Verify the arguments X# if ( $#ARGV != 1 ) { X print(STDERR "Usage: $zero system remote_dir\n"); X exit(1); X} X X$system = $ARGV[0]; X$remote_dir = $ARGV[1]; X X############################################################################# X# X# Build the batch ftp command. X# X$user = (getpwuid($<))[0]; # safe when run from "at". X$localhost = `hostname`; chop($localhost); if ( $localhost !~ /\./ ) { X # X # If the host name does not have '.' notation try to X # get an alias. We then hope it is in domain name notation. X # X @fullhost = gethostbyname($localhost); X $localhost = $fullhost[1] if $fullhost[1] ne ""; X} X$template = X" open $system X user anonymous ${zero}4$user@$localhost X bin X cd $remote_dir X ls -l X bye X"; X X############################################################################# X# X# Run the batch ftp command. X# open(CMD, "|$cmd") || die("Could not start command ($cmd). $!\n"); print CMD $template; close(CMD); X exit(0); END_OF_FILE if test 1853 -ne `wc -c <'ftpls.pl'`; then echo shar: \"'ftpls.pl'\" unpacked with wrong size! fi chmod +x 'ftpls.pl' # end of 'ftpls.pl' fi if test -f 'ftpmget.pl' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'ftpmget.pl'\" else echo shar: Extracting \"'ftpmget.pl'\" \(7128 characters\) sed "s/^X//" >'ftpmget.pl' <<'END_OF_FILE' X#!/bin/perl X X############################################################################# X# X# Copyright 1993 Scott Bolte (scott@craycos.com) X# X# Leave this copyright alone. But feel free to do with the script X# as you please. Sending me enhancements would be appreciated. X# X# If you feel like pretending this is shareware, and want to X# send some money my way, feel free. I promise not to object. X# X# Summary of ftpmget: X# X# Obtain a set of files via ftp from a remote system. Anonymous X# ftp is used. Can be asked to delay the command until a later X# time. Such a request will result in "at" being used. X# X# History: X# X# 1993.06.14 Initial implementation. X# X# 1993.06.15 Changed so that if a file contains white space X# it is obtained with get instead of mget. X# X# 1993.06.30 Added --at option to allow delayed operation. X# X# Examples: X# X# ftpmget hrd769.brooks.af.mil pub/FAQ FAQ_07 FAQ_09 X# X# cd ~/ftp/hrd769.brooks.af.mil X# ftpmget - pub/FAQ FAQ_07 FAQ_09 X# X# ftpmget hrd769.brooks.af.mil pub/FAQ - << EndOfList X# FAQ_07 X# FAQ_09 X# EndOfList X# X############################################################################# X X$zero = $0; X$zero =~ s,.*/,,; X sub usage { X X print <) { X chop; X $file = $_; X $file =~ s/^\s+//; # leading white space X $file =~ s/\s+$//; # trailing white space X $file =~ s/^"(.*)"$/$1/; # Remove enclosing quotes X push(@files, $file); X } X} X X# X# Make spaces safe for all mankind. N. Armstrong. X# foreach $file (@files) { X next if $file eq ""; # skip empty names X $file =~ s/(.*)/"$1"/ if $file =~ /\s/; # add quotes if need be. X push(@tmp, $file); X} X@files = @tmp; undef @tmp; X X############################################################################## X# X# If a delay was asked for run the command later. X# X if ( $delay_time ne "" ) { X $me = $0; X $me .= " --verbose" if $verbose; X $me .= " --debug" if $debug; X $me .= " $system $remote_dir - << End_Of_List\n"; X foreach $file (@files) { X $me .= "\t$file\n"; X } X $me .= "End_Of_List\n"; X X $at = "at $delay_time"; X X print(STDERR "At $delay_time the following command will be run:\n"); X print(STDERR " $me\n"); X if ( $debug ) { X print(STDERR "Skipping command in debug mode.\n"); X exit(0); X } X open(CMD, "|$at") || die("Could not run command ($at). $!\n"); X print(CMD $me); X close(CMD); X exit(0); X} X X############################################################################## X# X# Build the batch ftp command. X# X X$user = (getpwuid($<))[0]; # safe when run from "at". X$localhost = `hostname`; X chop($localhost); if ( $localhost !~ /\./ ) { X # X # If the host name does not have '.' notation try to X # get an alias. We then hope it is in domain name notation. X # X @fullhost = gethostbyname($localhost); X $localhost = $fullhost[1] if $fullhost[1] ne ""; X} X$template = X" open $system X user anonymous ${zero}4$user@$localhost X bin X prompt X cd $remote_dir X"; foreach $file (@files) { X if ( $file =~ /\s/ ) { X push(@space_files, $file); X next; X } X if ( length($line) + length($file) + 1 > 75) { X $template .= sprintf("$line\n"); X $line = ""; X } X $line = " mget" if $line eq ""; X $line .= " $file"; X} X$template .= "$line\n"; foreach $file (@space_files) { X $template .= " get $file\n"; X} X$template .= "$bye\n"; X X############################################################################## X# X# Run the batch ftp command. X# X open(CMD, "|$cmd") || die("Could not start command ($cmd). $!\n"); print CMD $template; close(CMD); X X############################################################################## X# X# Report whether or not the files were obtained. X# X print("\nReport for file transfers from $system.\n"); print(" Remote directory \"$remote_dir\".\n"); print(" Local directory \"$pwd\".\n"); foreach $file (@files) { X if ( -f $file ) { X print(" Obtained \"$file\".\n"); X next; X } X X if ( $file =~ /^".*"$/ ) { X $file =~ s/^"(.*)"$/$1/; X if ( -f $file ) { X print(" Obtained \"$file\".\n"); X next; X } X } X push(@again, $file); X} X X############################################################################## X# X# If there was a problem getting all the files print out a command X# that can try again later. X# X if ( $#again >= $[ ) { X @time = localtime($time); X $next = sprintf("%2d:%02d", $time[2], $time[1]); X X print <