One build, test, deploy pipeline
Part one pushed a commit and the running app changed on the spot, with no check standing between the push and the live code. This chapter puts a gate there. One pipeline runs three stages in order, build then test then deploy, and a red test stops the release before it ever reaches the running app. Everything runs on the same Ubuntu 24.04 VPS that already serves pipeclip, and each result below is what that machine returned.
- The pipeline checks the pushed commit out into a throwaway staging tree, so build and test run against a copy and a broken commit never replaces the live code.
- Only a green build and a green test reach the deploy stage; a failing test exits the script and the running app stays on the old version.
- A post-receive hook runs after the push, so it gates the release, not the push itself; refusing the push is a pre-receive hook's job.
- A post-deploy health check confirms the restarted service actually answers 200 before the pipeline is called a success.
The raw checkout from part one is fine until the day you push code that does not compile. Then the live app breaks at the moment of the push, and you find out from a user rather than from a test. A pipeline moves that discovery to before the deploy.
Prerequisites
- The git-hook deploy from part one, with a bare repo at
/srv/git/pipeclip.gitand the app under/srv/pipeclip. - The
pipeclipsystemd service running under thepipeciuser, answering on127.0.0.1:8093. - A
tests/directory in the repo with a fewtest_*.pyfiles, andrsyncinstalled. - Sudo rights for
pipecito restart the one service, scoped in sudoers.
The pipeline runs on a staging copy
The script lives at /srv/pipeclip/bin/pipeline.sh. It takes the pushed revision, checks it out into a fresh temp directory from mktemp -d, and runs build and test there. The live directory is untouched until both pass. set -euo pipefail means any failing command exits the whole script, so a red stage cannot fall through to the next one.
#!/usr/bin/env bash
set -euo pipefail
REV="${1:-main}"
GIT_DIR=/srv/git/pipeclip.git
LIVE=/srv/pipeclip/app
STAGE="$(mktemp -d)"; trap 'rm -rf "$STAGE"' EXIT
git --git-dir="$GIT_DIR" --work-tree="$STAGE" -c advice.detachedHead=false checkout -q -f "$REV"
cd "$STAGE"
# build
python3 -m py_compile app.py
# test
python3 -m unittest discover -s tests -p 'test_*.py'
# deploy (only reached if build+test passed)
rsync -a --delete "$STAGE"/ "$LIVE"/
sudo systemctl restart pipeclip
code="$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8093/health)"
[ "$code" = 200 ]
Three stages, read top to bottom. The build stage runs py_compile on app.py, a syntax and import check, and stamps the release with the short commit sha. The test stage runs unittest discover over tests/. Only when both return clean does the deploy stage rsync the staged tree onto the live path, restart the service, and confirm /health answers 200. The trap removes the temp directory whichever way the script ends.
Wire the hook to the pipeline
Part one's post-receive hook copied the checkout straight to the live directory. Now it hands the pushed commit to the pipeline instead. Read the hook back to see the change.
cat /srv/git/pipeclip.git/hooks/post-receive
The body is small: for each ref pushed, ignore anything that is not main, and pass the new commit to the pipeline script.
#!/usr/bin/env bash
while read -r _old new ref; do
[ "$ref" = refs/heads/main ] || continue
/srv/pipeclip/bin/pipeline.sh "$new"
done
One nuance matters, and it is easy to get wrong. A post-receive hook runs after the objects are already stored, so it cannot reject the push. The commit lands in the bare repo either way. What a failing test blocks is the release, not the push.
- The commit is written to
/srv/git/pipeclip.gitbefore the hook runs, so the push always succeeds. - A red test stops the pipeline at the test stage, so the deploy stage never runs and the live app is unchanged.
- To refuse the push itself on a red test, move the same build and test checks into a
pre-receivehook, which runs before the objects are accepted.
A red test stops the release
Time to prove the gate. Blank out greeting.txt, commit, and push to main as usual. The test suite includes test_greeting_not_empty, which reads that file and asserts it has content.
printf '' > greeting.txt
git commit -am "blank the greeting"
git push origin main
The push output shows the pipeline start, then the test stage fail. unittest prints .F., one dot per passing test and an F for the failed one, followed by the assertion message that greeting.txt must not be empty, then FAILED (failures=1). Because set -e is on, the script exits right there. The deploy stage does not run.
Confirm the live app for yourself. A curl to /version still returns 1.1.0, the version from before the push. The broken commit sits in the bare repo, but it never became the release. Nothing you would have to roll back has reached users.
A green run deploys the new version
Now fix the greeting and bump the version file, then push the same way.
printf 'Green pipelines ship.' > greeting.txt
printf '1.2.0' > VERSION
git commit -am "fix greeting; bump to 1.2.0"
git push origin main
This time all three stages pass. The build stage compiles cleanly and stamps short sha 83340a9. The test stage prints Ran 3 tests in 0.03s and OK. The deploy stage rsyncs the staged tree over, restarts the service, and the health check reports post-deploy health: 200, ending on deployed 1.2.0 build 83340a9. A curl to /version now returns 1.2.0. The release moved forward only because the gate was green.
That is the whole point of the staging tree. Build and test ran against a copy, so the red push failed loudly without touching the running code, and the green push replaced it only after both stages passed.
After the red push, curl -s http://127.0.0.1:8093/version still returns the old number and the push output ends in a failure. After the green push, the same curl returns the bumped version and the log ends in deployed. If a red push ever changes the live version, the deploy stage is not gated behind the test stage; check that set -e is on and the stages run in order.
Going further: reject the push, not just the release
This pipeline gates the release but still accepts every push, because a post-receive hook runs after the objects are stored. If you would rather the push itself fail on a red test, git gives you an earlier hook. A pre-receive hook runs before the commit is written to the bare repo, so exiting it non-zero refuses the push outright and the bad commit never lands in history at all.
The trade-off is where the failure shows up. A post-receive gate keeps history complete, which is handy for bisecting later, and blocks only the deploy. A pre-receive gate keeps the repo clean but rejects the developer's push, which can be jarring mid-flow. Many teams run the fast checks in pre-receive to catch obvious breakage and the fuller suite in the deploy pipeline. Whichever you pick, the staging-copy pattern is the same: never build or test in the directory the live service runs from.
Frequently asked questions
Does a failing test undo the commit that got pushed?
No. The commit is stored in the bare repo before the post-receive hook even starts, so a red test cannot remove it. The hook runs afterward and gates the release: it stops the pipeline before the deploy stage, so the live app stays on the old version. If you want the push itself rejected when a test fails, run the checks in a pre-receive hook, which fires before the objects are accepted.
Why build and test in a temp directory instead of the live path?
Because a compile error or a failing assertion should never reach the directory the service is running from. The pipeline checks the commit out into a mktemp -d tree and runs build and test there. Only a clean pass triggers the rsync onto /srv/pipeclip/app. If either stage fails, the temp tree is deleted by the trap and the running code is exactly as it was.
The tests already passed, so what does the health check add?
The tests check the code in isolation, before the service restarts. The health check confirms the restarted process actually binds and answers on 127.0.0.1:8093. It catches problems tests cannot see: a config the new version needs but the box does not have, or a port that failed to come up. If /health is not 200, the final line fails, the script exits non-zero, and the failure shows in the push output.
Can I add slower tests without dragging out every push?
Keep the fast unit tests in the gate so pushes stay quick. Run heavier suites separately, either as a later stage that only fires on a green gate, or on a nightly schedule off the push path. The discover pattern test_*.py picks up whatever you drop into tests/, so growing the fast set is just adding files. Reserve the gate for checks worth blocking a release over.
What you have, and what comes next
You have one pipeline that runs build, test and deploy in order on Ubuntu 24.04, with a staging copy so a broken commit fails safely. You proved both directions: a red test kept the live app on 1.1.0, and a green run shipped 1.2.0 with a matching build sha.
The deploy still restarts the service in place, which drops requests in flight during the bounce. The next chapter swaps the release to a new directory and flips a symlink, so the cutover is atomic and a live request never lands mid-restart.