Time Out for Timeouts
I’ve been working in a private GitHub repo for the CPAN Security Group. The account is a free org account and it has 2,000 minutes of GitHub Actions per month. The minutes are used for private repositories. Late last month we ran out of CI credits, so I took a closer look at the problem. It turns out there were two problems:
- some processes in CI had been hanging and burning through CI minutes without actually doing anything
- these jobs didn’t have an explicit timeout set
The GitHub docs for limits say:
Each job in a workflow can run for up to 6 hours of execution time. If a job reaches this limit, the job is terminated and fails.
Photo by Alfonso Scarpa on Unsplash.
#That’s a generous default
6 hours is a long time for a job to run. At a limit of 2,000 minutes you only need to max out your actions 6 times to exhaust your monthly minutes. I’m oversimplifying this a bit since minutes is an easy way to express it. The 2,000 is actually Linux 2-core minutes, with Windows and macOS minutes costing more. So if you’re not using only Linux 2-core then you’ll burn through your credits even faster.
There are a couple of remedies here.
- set reasonable timeouts in your own code to avoid hangs (you already do this, except in the cases where you don’t)
- set an explicit timeout in your CI jobs which is hopefully less than the very generous default setting
I will leave #1 as an exercise for the reader. #2 might look something like:
jobs:
test:
runs-on: ubuntu-latest
# Cancel the whole job if it runs longer than this. Don't go to 360.
timeout-minutes: 15
steps:
- uses: actions/checkout@v6 # you may want to pin to a hash here rather than a version
- name: Run the test suite
# You can also apply limits to a single step.
timeout-minutes: 10
run: prove -lr t
Did this help with the problem I was trying to solve? Yes and no. We’ve burned through our minutes again this month, so it’s time to look at the individual workflows and see where we can hammer down our usage. Your LLM can be useful here, querying your GitHub Actions logs to see where the time is going and making suggestions to optimize, like running more tests in parallel on machines that have the capacity.
Your LLM can also be useful in trying to find a reasonable timeout for your existing jobs, saving you a lot of poking around and also a bit of math.
Eventually you may just find it cheaper to offload your CI to a self-hosted runner. For My Mind is Racing, I’ve found it to be much more affordable to spin up my CI at Hetzner than directly at GitHub. It has saved me hundreds of dollars already and it wasn’t all that complicated to set up.
#Finding workflows without a timeout
I asked Claude to create a script which would look at my own repositories (not forks) and find jobs which don’t have timeouts. There are a lot of them. I’ll need to start chipping away at them.
The script lives in my dot-files as
bin/gh-missing-timeouts.
It requires:
Contains LLM-generated content not written by Olaf
#!/usr/bin/env bash
set -euo pipefail
usage() {
cat <<'EOF'
Usage: gh-missing-timeouts [--skip N] [--limit N] [owner | owner/repo]
List GitHub Actions jobs that don't set timeout-minutes, so they fall back
to the 360-minute default. Prints one line per job: repo/path: job
Arguments:
owner User or org whose repos to scan (default: you)
owner/repo Scan just this one repo
Options:
--skip N Owner scans only: skip the first N repos (default: 0)
--limit N Owner scans only: maximum repos to scan (default: 1000)
-h, --help Show this help
When scanning an owner, repos are sorted by name, so --skip and --limit
page through them in a stable order. Forks and archived repos are left
out before paging. Jobs that call a reusable workflow (job-level `uses:`)
are always skipped, since those can't set timeout-minutes.
Requires gh (logged in), yq (tested with the Python yq, kislyuk/yq)
and is (https://github.com/oalders/is).
Examples:
gh-missing-timeouts
gh-missing-timeouts --limit 50 my-org
gh-missing-timeouts --skip 50 --limit 50 # repos 51-100
gh-missing-timeouts oalders/my-mind-is-racing
EOF
}
die() {
echo "gh-missing-timeouts: $1" >&2
echo "Try 'gh-missing-timeouts --help'." >&2
exit 2
}
owner=
limit=1000
skip=0
while (($#)); do
case $1 in
-h | --help)
usage
exit 0
;;
--skip)
(($# >= 2)) || die "--skip needs a number"
skip=$2
shift 2
;;
--skip=*)
skip=${1#--skip=}
shift
;;
--limit)
(($# >= 2)) || die "--limit needs a number"
limit=$2
shift 2
;;
--limit=*)
limit=${1#--limit=}
shift
;;
-*)
die "unknown option: $1"
;;
*)
[[ -z $owner ]] || die "only one owner or owner/repo allowed"
owner=$1
shift
;;
esac
done
[[ $limit =~ ^[1-9][0-9]*$ ]] || die "--limit must be a positive integer, got '$limit'"
[[ $skip =~ ^[0-9]+$ ]] || die "--skip must be a non-negative integer, got '$skip'"
command -v is >/dev/null || die "is is required but not installed"
for cmd in gh yq; do
is there "$cmd" || die "$cmd is required but not installed"
done
# Job and file names come from the scanned repos and can hold terminal escapes.
print_safe() {
printf '%s\n' "${1//[[:cntrl:]]/?}"
}
# Jobs that call a reusable workflow (job-level `uses:`) can't set
# timeout-minutes, so they're skipped.
filter='.jobs // {} | to_entries[]
| select(.value.uses == null and .value["timeout-minutes"] == null)
| .key'
list_repos() {
if [[ $owner == */* ]]; then
# Fails loudly if the repo doesn't exist or isn't visible.
gh repo view "$owner" --json nameWithOwner --jq .nameWithOwner
else
# gh repo list has no offset, and its order can shift between runs,
# so fetch every repo, sort by name, then slice out the page.
local repos
repos=$(gh repo list ${owner:+"$owner"} --source --no-archived \
--limit 100000 --json nameWithOwner --jq '.[].nameWithOwner') || return
LC_ALL=C sort <<<"$repos" |
awk -v skip="$skip" -v limit="$limit" \
'NF && NR > skip && NR <= skip + limit'
fi
}
repos=$(list_repos)
status=0
err=$(mktemp "${TMPDIR:-/tmp}/gh-missing-timeouts.XXXXXX")
trap 'rm -f "$err"' EXIT
while read -r repo; do
[[ -n $repo ]] || continue
if ! paths=$(gh api "repos/$repo/contents/.github/workflows" \
--jq '.[] | select(.name | test("\\.ya?ml$")) | .path' 2>"$err"); then
# 404 means the repo has no .github/workflows directory.
grep -q 'HTTP 404' "$err" && continue
echo "$repo: $(<"$err")" >&2
status=1
continue
fi
while read -r path; do
[[ -n $path ]] || continue
# The raw media type also covers files over the JSON API's 1MB limit.
if ! jobs=$(gh api -H 'Accept: application/vnd.github.raw+json' \
"repos/$repo/contents/$path" | yq -r "$filter"); then
print_safe "$repo/$path: could not fetch or parse" >&2
status=1
continue
fi
while read -r job; do
[[ -n $job ]] || continue
print_safe "$repo/$path: $job"
done <<<"$jobs"
done <<<"$paths"
done <<<"$repos"
exit "$status"
Related posts:
