Strange contents on some ftp mirrors

[Marcin Wisnicki]

On Wed, 28 Jul 2010 17:11:25 +0200, Julian H. Stacey wrote:

> Marcin Wisnicki wrote:
>> 
>> Alternatively it could be prevented with no disk cost by putting
>> "MIRROR-IN-PROGRESS" file and making all package utilities check for
>> its existence and fail with explanation to try again later.
> 
> Perhaps a less visible &/or top listing semaphore of
> ./.MIRROR-IN-PROGRESS ? Content could be empty, or mirror master could
> generate into it, eg:
> 	Version:	Number of format of .MIRROR-IN-PROGRESS\n Master-URL:	Of
> 	master\n
> 	Protocol:	Of mirror\n
> 	Date: 		(GMT) start of mirror\n
> 	Contact:	If mirror fails\n
> 	Text:		Any other text.\n
> 	EOR:		End of record\n
> 
> Various FreeBSD etc stuff gets mirrored as nested directories, so one
> might want to to be able to check up to N steps up the parent tree
> structure for lock(s).
> 

Unfortunately no scheme involving global lock files will really work.
I've read http://wiki.freebsd.org/FreeBSDMirrors that you linked and it mentions
TIMESTAMP files that it uses to monitor mirrors quality.

You just need to visit $random[1] freebsd mirror and look at that file
then check packages directory. You will find that it can have latest
timestamp yet many months old files.
In other words, mirrors cannot be trusted.

For now however I've found and suggested in sibling post a workaround
that should mostly work assuming INDEX file gets mirrored before packages
(and it looks like it is).

[1] random = ftp.pl.freebsd.org, but it's not the only one

> One might also want an optional force- fetch- regardless flag.
> 
> Awkward conditions to allow for:
>   What if mirror is not mirroring when you start your fetch, but is when
>   you finish ?
>   What if mirror is not mirroring when you start your fetch, then mirror
>   starts & finishes a mirror before you finish your fetch ?
> 

Solving any of this, in a way that does not require intervention of user if it
happens, requires that file content at given url is immutable (or doesn't exist).
There is simply no other 100% reliable way.

You can however do some sanity checks and fail early if failing is acceptable.
Some ideas:
1. if (mtime(pkg) > mtime(INDEX)) fail("Mirror sync in progress");
2. Fetch timestamps of all requested packages in one go before fetching contents;
   do it again and compare to be sure nothing changed;
   fetch actual packages and fail if mtime/size does not match.
3. Do 1 or 2 but fetch timestamps from well known source or multiple sources.
   You may also find most up-to-date mirror this way.

> One might want to generalise a mechanism that could be added to multiple
> tools, eg  ports/net/ mirror/ rdist6/ rsync/ ?...
>