Team project, Columbia University, May 2026
ELI5: Explain Like I'm 5
A news pipeline that turns dense articles into plain-language summaries.
From headline to plain English
- News APIs
- MongoDB Atlas
- PySpark
- PostgreSQL
- AI summaries
- Web app
In brief
- The question
- News is fast, dense and full of jargon. How do you turn thousands of articles a day into something anyone can understand?
- The approach
- A data pipeline that pulls articles from three news APIs, processes them with Spark, stores them in a cloud database, and generates simple summaries with an AI model, served through a web app.
- The result
- A working app processing over 20,000 articles from three sources, with pre-generated summaries, a switch back to the original article, and follow-up questions answered by searching across articles.
The idea
Most readers scroll past articles they don't fully understand. ELI5 takes real-time news from three sources and simplifies it: read the plain version, switch to the original, or ask a follow-up question.
How the pipeline works
- Ingestion from three news APIs: The Guardian, GNews and NewsData.io.
- Raw articles land in MongoDB Atlas, whose flexible schema copes with inconsistent JSON from different sources.
- PySpark on Databricks standardizes the articles and loads them into PostgreSQL on Supabase, whose structured schema makes filtering by category, date and source fast.
- Claude Haiku writes the plain-language summaries, and Pinecone, a vector database, supports search for follow-up questions.
- A Flask front end displays the summarized articles.
Decisions and why
- Each store was chosen for its job: a document database for messy raw data, a relational one for fast filtered reads, a vector one for semantic search.
- The team weighed the cost of each choice too: Spark is more than a small dataset needs but scales, and per-call AI costs add up at volume.
- A production version would add reading preferences, engagement data and subscriptions to personalize the feed.