S3-backed CI cache helper. A single static Go binary.
cacher is the build that the typical "check S3, fall back to
upstream, upload for next time" shell loop should have been. It downloads,
uploads, lists, and invalidates cached artifacts in any S3-compatible
bucket — single files, docker images (via streamed
save | zstd | s3), and whole directory trees keyed by lockfile
hash. Built for builds.sr.ht; works
anywhere you can run a binary and reach an S3 endpoint.
Pre-built binaries for v0.2.2. The shipping URL is stable; the SHA-256s below are specific to this build — paste them into your pin if you care.
| Platform | Binary | SHA-256 |
|---|---|---|
| linux/amd64 | cacher-linux-amd64 | 310165e2765d9834fc8b56011d4e4b4fb49805722dc674a4e65c506b51d4e168 |
| linux/arm64 | cacher-linux-arm64 | 6490a9b7ecc7a5bf843ec5d49251bbd97a89cad8e8837233e5d0f7d45698c33d |
| darwin/amd64 | cacher-darwin-amd64 | 323176428bf2e53c3f81bcfafe03252d9692b1dc97b91c592d51a26b53c5b3bd |
| darwin/arm64 | cacher-darwin-arm64 | 8300cc4c27f5529164f2ef5a3a788cd140ae86b26f8fef103900d2335aa9d4bb |
The install.sh bootstrap picks the right asset for
the host, verifies it against checksums.txt, and
installs into ~/.local/bin — on builds.sr.ht it also appends the
PATH export to ~/.buildenv, so a CI manifest needs
exactly one line:
curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/install.sh | sh
By hand, all four hashes plus filenames are also available in a single
checksums.txt for piping into sha256sum -c:
wget https://bigbes.pages.srht.bigb.es/ci-cacher/cacher-linux-amd64 \
-O ~/.local/bin/cacher
chmod +x ~/.local/bin/cacher
wget -qO- https://bigbes.pages.srht.bigb.es/ci-cacher/checksums.txt \
| sha256sum -c --ignore-missing
# before — install awscli, write ~/.aws/config, then in every task:
if aws s3api head-object --bucket "$B" --key "$K" >/dev/null 2>&1; then
aws s3 cp "s3://$B/$K" "$out"
else
curl -sSL "$url" -o "$out"
aws s3 cp "$out" "s3://$B/$K"
fi
# after — one binary, one config, one command:
cacher download "$key" "$out" --url "$url"
init / doctor — persist config + smoke-test credsdownload / upload — single file, with --url fallback and --sha256 verifyexists / list / delete / key — management + shell helpersdocker {exists,download,upload} — streamed save/load via zstddir {download,upload} — tar+zstd directory caching, keyed by content hash--exec on every download — run a build script on a miss
and seed the cache with what it produced; --optional makes a miss exit 0
Run cacher --help for the full surface. Read
the
README for usage patterns, Garage compatibility notes, and exit-code semantics.
The shell version this replaced repeated five things in every CI task:
install AWS CLI v2 (≈50 MB per build), write a Garage-tuned
~/.aws/config, compute cache keys from file content with
sha256sum | cut, branch HIT/MISS by hand, and for
docker images pipe docker save | zstd | aws s3 cp - (and the
inverse). cacher collapses all of that into one fetched binary
plus a config file. The directory caching is the genuinely new capability —
the shell version only ever handled single files.
All notable changes to this project will be documented in this file. The format is based on Keep a Changelog and this project adheres to Semantic Versioning.
dir download could not restore a tree containing read-only
directories — which the Go module cache is throughout (0555 dirs,
0444 files). Extraction applied each archived mode as the entry
landed, so the first file written into a restored 0555 directory
failed with permission denied. Directories are now created
writable and their recorded mode applied once extraction finishes,
files are written writable and chmod'ed afterwards, and a read-only
file left by an earlier restore is replaced instead of erroring.
Caught by this repo's own CI on the first cache HIT of the new
gomod/*.tar.zst key (build #295).install.sh verified the download with sha256sum -c --ignore-missing,
a coreutils extension busybox rejects — so on Alpine, which is what
several manifests using this run on, the bootstrap aborted with a bogus
"checksum mismatch". The hash is now compared as a plain string, which
busybox, coreutils and macOS shasum all handle.install.sh stages the binary inside the target directory and renames
it into place, so installing over a cacher that is currently running
can't fail with ETXTBSY.--exec <script> on download, dir download and docker download —
on a cache miss the script runs through sh -c and its result seeds
the cache. Generalises the --url and --pull fallbacks to anything
expressible as a shell command, which collapses the
restore-or-build if/fi block CI manifests repeat per cache:
cacher dir download "gomod/{hash}.tar.zst" ~/go/pkg/mod \
--hash-from go.sum --exec 'go mod download'
The destination directory is created before the script runs; the seed upload is best-effort; a failing script is fatal and propagates its own exit status.
--optional on download and dir download — a cache miss exits 0
instead of 1/3, so set -e manifests no longer need a trailing
|| echo "cache miss".
install.sh, published next to the binaries: detects the platform,
verifies against checksums.txt, installs into ~/.local/bin and, on
builds.sr.ht, appends the PATH export to ~/.buildenv. Reduces the
bootstrap task to
curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/install.sh | sh.
max-width from 44rem to 55rem.cacher docker download --pull — on cache miss, falls back to
docker pull <image:tag> and seeds the S3 cache. Mirrors the
--url fallback on file download; collapses the if/else
cache-or-pull bash dance in CI manifests to a single command.First public release. Replaces the s3_cache_or_curl /
s3_cache_docker_image shell helpers in tarantool-protobuf/.builds/lib/ci-lib.sh
with a single static Go binary.
cacher init / cacher doctor — persist config to
~/.config/cacher/config.toml and smoke-test S3 credentials (HEAD bucket
cacher download / upload / exists / list / delete for single
files. Download falls back to --url on cache miss and back-fills the
cache. Optional --sha256 verifies the fetched content.cacher docker {exists,download,upload} — streamed docker save | zstd
→ S3 multipart upload (and inverse). Pure-Go zstd via
klauspost/compress, no external
zstd binary on the host.cacher dir {download,upload} — tar+zstd of a directory tree keyed by
content hash. Closes the gap left by the shell version, which only
cached single files.cacher key — resolve a key template (substituting {hash}) for shell
scripting.--hash-from <path> (repeatable; files or directories) on every command.
For a single file path the digest exactly matches sha256sum file | cut -c1-N,
so existing keys migrate without recomputation.--arch-suffix opt-in to suffix every key with -<goos>-<goarch>.list --recursive for flat listing; list --root to ignore the
configured prefix and list at bucket root. Default output style mirrors
aws s3 ls (delimited by /).when_required (Garage
doesn't implement boto3 1.36+ trailing CRC32 checksums).go test -tags=e2e) against a real Garage
container via testcontainers-go using dxflrs/garage:v2.3.0's
--single-node --default-bucket mode.unit.yml (every push), e2e.yml (every push, with
Docker), publish.yml (tags only, ships cross-platform binaries via
goreleaser to pages.sr.ht and as build artifacts).doctor uses ListObjectsV2 (1-key) instead of HeadBucket — Garage
rejects HeadBucket with 403 even for valid credentials.~/.config/cacher/config.toml on every platform
(previously fell into ~/Library/Application Support/cacher on macOS
via os.UserConfigDir). cacher is a CI tool; dev-macs and Linux runners
must look in the same place.