Been dealing with this more often lately. Tests pass on my machine, I push, and CI blows up. Usually it’s one of these:
- Different Node/Python/whatever version
- Missing env vars that exist in my .env but not in CI secrets
- File system case sensitivity (macOS vs Linux)
- Some flaky test that depends on timing
My current debugging flow is pretty basic: check the logs, compare versions, run the exact same Docker image locally if I can. But it still eats 20-30 minutes each time before I figure out the actual problem.
Anyone have a more systematic approach? Like a quick checklist you run through before you even look at the logs?
Also curious — do you replicate your CI environment locally with something like act (for GitHub Actions) or just trust the remote runner?
Whatever the logs indicate is the issue? How can there be a fixed response to this?
You should definitely use your builder container to run your tests locally before pushing changes though. That’s one of the advantages of containers - a consistent environment.
How viable moving over to carpentry or metal working would be.
Lately, we’ve been use Justfiles to setup the env and run the tests. The ci just runs the just recipes.
Makes replicating on the dev machine much easier.
I use mise-en-place for this same sort of thing. Many of my shared workflows demand a certain contract be satisfied including things like providing artifact coordinates, mise tasks (build, publish, etc.). One can also use mise to specify the runtime versions to simplify local build and test and ci using the same runtimes.
Of course, there is no best solution, since this is a “how long is a piece of string” sort of question.
If something fails look at the logs, actually READ them, what does it sail failed? Investigate that.
Failure + no logs?
Your mission is now to first find a way to enable logging, then read them.
No way to enable logging? If the infra and code is yours, now you need to write in logging, enable it, and then read the log
Not your code? No logs?
Figure out the difference between the last working version of your code / deployment script and the current one. If no difference check the change log for the infra you are using
That’s my brief list
I first check the logs for what the failure is.
In QEMU all of our CI environments are replicable locally as docker/podman images. However usually flaky CI is due to races exposed on overloaded runners so I often also run make -j(nproc) at the same time to simulate that. A retry script is also useful to get an idea of how stable a test is. Having sanitizer builds can also help.
Usually, when CI fails, it’s a flaky CI failure or that I have in fact not run it locally.
I guess some context is missing. My build flow seems much simpler. I check the CI steps overview and step log, and then I know what’s wrong. That doesn’t take 20 min. More like ~3 min.
At work, we use Jenkins and the runners are owned infrastructure. If I debug what goes wrong in the CI environment, I go into the pipeline definition, and do the calls locally, just like that, on the command line. No need for complicated environment replication beyond that.
dotnet restore,dotnet build --no-restore,dotnet package --no-build,dotnet test --no-build, etc.If it’s not CI specific, issues show up in the normal local tooling without special env prep.
do you replicate your CI environment locally with something like act (for GitHub Actions)
this is what does it for me most times; but I usually also have a CI .env file
actcan use and usejustto abstract recipes e.g. when runningjust test, either CI or local will run depending if the CI env var is set. It’s the same battery of tests, only different env files.
