new failure on hobson9; immediately stopped

building de-freebsd-doc-20090913 on hobson9.isc.freebsd.org
build started at Fri Oct 30 09:29:26 UTC 2009 (1256916566.0)
[...]
tar: Retrying...
tar: Damaged tar archive: Invalid argument
[another 6666 instances]
tar: bzip2 decompression failed
tar: Error exit delayed from previous errors.
pkg_add: tar extract of /tmp/depends/teTeX-texmf-3.0_6.tbz failed!

on pointyhat:

$ ls -al teTeX-texmf-3.0_6.tbz
-rw-r--r--  1 ports-amd64  4206  82724827 Oct 14 17:37 teTeX-texmf-3.0_6.tbz
$ du -sm teTeX-texmf-3.0_6.tbz
80      teTeX-texmf-3.0_6.tbz
from logs: all last-modified entries = 1255541875 (2009-10-14 17:37:55)

last 2 entries in squid:

1256894848.865 (2009-10-30 09:27:28.865000)

looking at *-freebsd-doc for last built times on hobson9:

building pl-freebsd-doc-20090913 on hobson9.isc.freebsd.org
build started at Fri Oct 30 08:40:25 UTC 2009
build of /usr/ports/misc/freebsd-doc-pl ended at Fri Oct 30 12:08:47 UTC 2009

building nl-freebsd-doc-20090913 on hobson9.isc.freebsd.org
build started at Fri Oct 30 09:09:04 UTC 2009
build of /usr/ports/misc/freebsd-doc-nl ended at Fri Oct 30 12:05:18 UTC 2009

both of these are correct builds.

later:

# date
Sat Oct 31 00:34:54 UTC 2009
# df -m
/dev/da0s1d                            103867 22206 73351    23%    /usr2
/dev/md2                                   30     0    27     2%    /var
/dev/md3                                   19     0    17     0%    /tmp

# cache_mem 8 MB
# cache_swap_low 90
# cache_swap_high 95
maximum_object_size 400 MB
# maximum_object_size_in_memory 8 KB
cache_dir ufs /usr2/squid/cache 6000 16 256 (that's 6G?)
# debug_options ALL,1

hobson9# du -sm *
5395    cache
1024    logs

XXX question: why does the build time seem to be later than any
log entries?

# ls -al 00/04/000004FF
-rw-r-----  1 squid  squid  82725389 Oct 30 09:27 00/04/000004FF

editing out header stuff, on freefall:
$ ls -al
-rw-r-----  1 linimon  devel  82725389 Oct 31 01:56 000004FF
-rw-r--r--  1 linimon  devel  82724828 Oct 31 01:56 foo
$ mv foo foo.tbz
$ bunzip2 foo.tbz
bunzip2: foo.tbz: trailing garbage after EOF ignored
the last N lines of tar -tvf appear ok

freefall:
216719360 Oct 31 01:56 foo.tar
pointyhat:
216719360 Oct 31 02:00 teTeX-texmf-3.0_6.tar

-------------------------------------------------------------------

comparing to gohan11, which does not experience this problem:

# cache_mem 8 MB
# cache_swap_low 90
# cache_swap_high 95
maximum_object_size 400 MB
# maximum_object_size_in_memory 8 KB
cache_dir ufs /x/squid/cache 4000 16 256
# debug_options ALL,1

other than 4G vs 6G, this is identical to hobson9.

gohan11# du -sm *
3626    cache
7951    logs

gohan11# date
Fri Oct 30 17:45:27 PST 2009
gohan11# df -m | head -10
devfs            0     0      0   100%    /dev
/dev/md0         4     2      1    53%    /etc
/dev/ad4a    20384 13754  96999    12%    /x
/dev/md1        30     0     28     1%    /var
/dev/md2        19     0     17     0%    /tmp