[Bug 241148] Mk/Scripts/do-fetch.sh: script made by fetch-list fetches the same file several times
bugzilla-noreply at freebsd.org
bugzilla-noreply at freebsd.org
Wed Oct 9 09:52:20 UTC 2019
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241148
Bug ID: 241148
Summary: Mk/Scripts/do-fetch.sh: script made by fetch-list
fetches the same file several times
Product: Ports & Packages
Version: Latest
Hardware: Any
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: Individual Port(s)
Assignee: rust at FreeBSD.org
Reporter: brigadir15 at gmail.com
CC: portmgr at FreeBSD.org
Flags: maintainer-feedback?(rust at FreeBSD.org)
Assignee: rust at FreeBSD.org
After r507705[1] fetch-list for ports with DISTFILES having entries with
subdirectories, produces shell script that causes downloading of the same file
several time.
For example:
$ cd /usr/ports/lang/rust
$ make fetch-list
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { env
/usr/bin/fetch -Fpr -S 96163304
https://static.rust-lang.org/dist/rustc-1.38.0-src.tar.xz || env
/usr/bin/fetch -Fpr -S 96163304
http://distcache.FreeBSD.org/ports-distfiles/rust/rustc-1.38.0-src.tar.xz ||
echo "rustc-1.38.0-src.tar.xz" not fetched; }
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkdir
-p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
https://static.rust-lang.org/dist/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
|| echo "2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz" not fetched; }
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkdir
-p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 87956779 -o
2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
https://static.rust-lang.org/dist/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 87956779 -o
2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 87956779 -o
2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 87956779 -o
2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 87956779 -o
2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 87956779 -o
2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz
|| echo "2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz" not
fetched; }
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkdir
-p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 6712004 -o
2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
https://static.rust-lang.org/dist/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 6712004 -o
2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 6712004 -o
2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 6712004 -o
2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 6712004 -o
2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 6712004 -o
2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz
|| echo "2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz" not fetched; }
Execution of that script ends up with this result:
$ su root -c "/usr/home/user/fetch_rust.sh"
Password:
fetch: https://static.rust-lang.org/dist/rustc-1.38.0-src.tar.xz: size unknown
fetch: https://static.rust-lang.org/dist/rustc-1.38.0-src.tar.xz: size of
remote file is not known
rustc-1.38.0-src.tar.xz 91 MB 4241 kBps 22s
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd 67 MB 3724 kBps 08s
fetch:
http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
2019-08-15/rust-std-1.37.0-x86_64-unknown-free 83 MB 4212 kBps 21s
fetch:
http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz:
Not Found
2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd 6554 kB 3455 kBps 02s
fetch:
http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz:
Not Found
fetch:
http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz:
Not Found
As you can see, the script tries to download, for example,
rustc-1.37.0-x86_64-unknown-freebsd.tar.gz five times. Even after the script
has downloaded it successfully on the first try.
That happens because of the following lines in fetch-list's script:
mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkdir
-p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
https://static.rust-lang.org/dist/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
|| mkdir -p "2019-08-15" &&
env /usr/bin/fetch -Fpr -S 70318740 -o
2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
|| echo "2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz" not fetched; }
It makes "2019-08-15", downloads (successfully)
rustc-1.37.0-x86_64-unknown-freebsd.tar.gz from
https://static.rust-lang.org/dist/2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz
(having zero exit status), skips `mkdir -p "2019-08-15"` (because of "no
errors" exit status), and fetches the file again after `&&` (once again --
because of zero exit status). This `fetch` is subsequently failing, but
`mkdir` after that `fetch` succeeds (because it had `-p` specified). The
script `fetch`es again. And so on.
I believe `mkdir` and `fetch` must be grouped together between all those `||`
operators.
See also: bug 239293[2]
P.S. If someone will fix this, please also take bug 239761[3] into account,
where my patch allows to combine several fetching lines into single one, as
usual.
[1] https://svnweb.freebsd.org/ports?view=revision&revision=507705
[2] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239293
[3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239761
--
You are receiving this mail because:
You are the assignee for the bug.
More information about the freebsd-ports-bugs
mailing list