Archive note, November 6, 2025: This post is based on SoCalNomad newsfeed deployment and feed-reliability documentation from October 2025.
AI was never the first hard problem in the SoCalNomad newsfeed.
The first hard problem was getting ordinary RSS feeds into a database reliably.
That sounds less interesting than clustering, summarization, or automated publishing. It was also more important. If the intake layer is unreliable, every clever downstream step inherits bad assumptions.
The 54-Feed Problem
The project collected Southern California entertainment and local culture sources from across the region. The goal was broad coverage without losing geographic focus.
The feed list included publishers with different RSS behavior, response times, formatting habits, and failure modes. Some feeds were fast. Some timed out. Some changed structure. Some were blocked by protection layers.
The production target was not perfection. It was reliable enough intake with clear attribution and deduplication.
That meant the ingestion workflow needed to preserve source identity, normalize article fields, hash URLs or titles for deduplication, and upsert into PostgreSQL without making a mess.
Attribution Was A Production Requirement
One early success criterion was 100% source attribution.
That was the right obsession.
If the system cannot say where an item came from, it cannot responsibly use that item. Attribution is not just a footer link. It is debugging information, editorial context, and legal hygiene.
For an automated newsroom, source attribution is infrastructure.
Batching Was Not An Optimization
RSS ingestion was batched to avoid making the workflow fragile. Too much work in one pass makes failures harder to isolate. Too little structure makes the system slow and noisy.
Batching gave the system a rhythm:
- Build the feed list.
- Process a manageable group.
- Wait briefly.
- Normalize results.
- Generate deduplication keys.
- Write to the database.
That sounds mechanical because it should be. Intake should be boring.
Bad Feeds Are Part Of The System
Some feeds will fail. Some publishers will block requests. Some URLs will age out. Some sources will stop being useful.
The documentation treated this as an operational reality rather than a surprise. A feed system needs maintenance: success rates, retry logic, removal decisions, and a willingness to stop treating every source as equally valuable.
That was another quiet production lesson. A data pipeline is not finished when it works once. It is finished enough when it can degrade visibly.
AI Needs Clean Inputs
The downstream AI steps depended on intake quality.
Relevance scoring, entity extraction, clustering, and synthesis all assume the system knows what article it is processing and where that article came from. If ingestion duplicates records or loses source names, the AI layer becomes a confusion amplifier.
That is why the boring feed work came first.
Before asking a model to write anything, the system needed to know what it had read.
What This Taught Me
Feed reliability is not glamorous. It does not produce a screenshot that looks like a product launch.
But it is one of the differences between a demo and a production pipeline.
A demo can paste three URLs into a prompt and get a nice paragraph.
A production system has to wake up tomorrow, fetch from dozens of imperfect sources, skip what failed, deduplicate what repeated, preserve attribution, and leave enough evidence for the next step to trust it.
SoCalNomad’s automated newsroom began there.
Not with AI magic, but with reliable intake.