Archive note, October 2, 2025: This post is based on the September 27, 2025 Docker networking troubleshooting notes.
One of the earliest SoCalNomad automation failures looked like an AI problem.
It was not.
The workflow was supposed to let n8n call a local LLM service running in another Docker container. The model was up. n8n was up. Both containers were attached to the same Docker network. The HTTP request node looked reasonable. The workflow simply ran until it timed out.
That is a particular kind of frustration: everything visible appears configured, but the box stays black.
The Symptom Was Not The Cause
The debugging path started at the application layer.
First there was an HTTP method problem. A request that should have been POST was sent as GET.
Then there was a body-format problem. The API expected JSON, not form-like parameters.
Both were real bugs. Fixing them still did not make the workflow work.
The remaining failure was a timeout, and timeouts are where bad assumptions hide.
The Containers Could Not Name Each Other
The key test was not in the n8n UI. It was inside the n8n container:
curl http://llm-service:8000/The service name did not resolve.
Both containers were on the same Docker network, but the n8n container was using an external DNS resolver instead of Docker’s internal DNS. It could look outward, but it could not resolve the service names Docker normally provides inside a bridge network.
The workaround was direct container IP communication. That proved the model server was reachable and the OpenAI-compatible endpoint worked.
The final lesson was more useful than the workaround: a container network is not magic. It is a set of assumptions that can be inspected.
Why This Mattered Later
That small failure shaped later infrastructure thinking.
SoCalNomad eventually had several kinds of boundaries:
- Browser to Cloudflare.
- Cloudflare to Apache.
- Apache to WordPress.
- WordPress to an internal API proxy.
- Automation workflows to PostgreSQL.
- Calendar frontend to calendar API.
- AI workflows to model providers.
At each boundary, the same question mattered:
Can the caller resolve the target, reach the target, speak the expected protocol, and receive the expected shape of response?
That sounds basic. It is also most of production debugging.
Black Boxes Need Inspection Ports
The phrase “black box” can mean a sealed mystery. That is not what I wanted.
For SoCalNomad, a good black box meant the public did not need to know how many small systems lived behind the site. It did not mean I could afford to be blind internally.
Every private component needed a way to answer simple operational questions:
- Is it running?
- Can another component reach it?
- Is DNS resolving the expected name?
- Is the port open?
- Is the response syntactically valid?
- Is the failure in transport, authentication, data, or logic?
The Docker DNS issue was the first reminder that architecture diagrams are promises, not evidence.
Evidence comes from curl, logs, shell access, and tests from the caller’s point of view.
The Humbling Part
I would like every production lesson to sound sophisticated. This one was not.
A workflow timed out because a container could not resolve another container’s name.
But that is exactly why it was useful. Production systems fail through ordinary details. A wrong DNS resolver. A stale cache. A missing environment variable. A service that restarts but does not pick up new code. A route that works from the host but not from inside the container.
The black box only works if the inside is boring enough to inspect.