How it works  ·  The agents  ·  Setup  ·  Customize  ·  FAQ

◈ The Idea

Most news is noise. Sift AI reads the day's candidates and hands you exactly one article worth your time, complete with key takeaways and a word of the day to grow your vocabulary.

Every day at the time you choose, a scheduler triggers a GitHub Actions workflow, which launches a Kaggle notebook. Inside it, a small crew of AI agents running open-source Qwen models collects fresh stories, picks the strongest one, summarizes it, chooses a useful English word, and emails everything to you as a clean PDF.

No server stays on. No LLM API is billed. Nobody has to press a button.

Daily At the time you choose

1 Article Chosen from ~20 candidates

4 Agents Select · Summarize · Vocab

$0 Inference cost

◈ What Lands In Your Inbox

A clean, print-ready PDF, every morning The full article, scraped and cleaned, word for word Key takeaways, 3 to 5 concrete facts pulled from the text Word of the day, an everyday English word with its meaning, an example and a practice scenario Source link back to the original publication The article is never rewritten by an AI. The body passes through a deterministic scraper and cleaner untouched. Models are only used for judging, summarizing and teaching.

◈ How It Works

Step	What happens
1	A scheduler (cron-job.org) calls GitHub's repository_dispatch API at the time you set. The workflow pushes the notebook to Kaggle with the CLI
2	Kaggle starts a T4 GPU kernel. The notebook installs Ollama and pulls the two Qwen models
3	It queries NewsData for each of your topics (globally and India-scoped) and reads editorial RSS feeds, then scrapes and cleans every candidate
4	The Senior News Editor agent picks the one article that best rewards a reader's time
5	Two lighter agents extract takeaways and choose a word of the day
6	A fixed HTML template is rendered to PDF with WeasyPrint and sent through Gmail

◈ Architecture

◈ The Agents

Two models, four agents. The bigger model does the one job that needs real judgment. The smaller one handles tasks on a single, already-clean input.

Agent	Model	Job
Senior News Editor	Qwen 2.5 · 7B Instruct	Compares all candidates and picks exactly one, or declares that none qualify
Content Summarizer	Qwen 3 · 1.7B	Extracts 3 to 5 specific takeaways, never inventing a claim
Vocabulary Word Selector	Qwen 3 · 1.7B	Picks one common, everyday English word, ignoring the article's topic
English Vocabulary Coach	Qwen 3 · 1.7B	Explains that word with a meaning, example and practice scenario. It never sees the article

How the editor chooses

The editor scores candidates in this order of priority:

1. Factual substance and depth
2. Credibility of the source and reporting
3. Learning value: a real argument, the "why" and not just the "what", and precise vocabulary
4. Real-world importance
5. Recency, only as a tie-breaker

Hard filters remove listicles, clickbait, press releases and vague "experts say" sourcing. Opinion and news are judged equally, with no built-in favorite.

What happens on a bad news day

The pipeline never forces a weak pick and never fails silently.

Attempt	Minimum length	Behavior
1	220 words	Strict bar. The editor may reject everything
2	150 words	Looser bar, topic match no longer a reason to reject
3	120 words	Last chance. Picks the best acceptable candidate
Fallback	n/a	No LLM call. Picks the best known-reliable, longest candidate

The article is scraped once. Retries only re-run the cheap selection step.

◈ Setup

1. Clone and point to your Kaggle account

git clone https://github.com/bytebymanas/sift-ai.git
cd sift-ai

Edit kernel-metadata.json and replace the username:

{
  "id": "YOUR_KAGGLE_USERNAME/sift-ai-daily-runner",
  "title": "Sift AI Daily Runner",
  "code_file": "sift-ai.ipynb",
  "language": "python",
  "kernel_type": "notebook",
  "is_private": true,
  "enable_gpu": true,
  "enable_internet": true
}

2. Add your secrets in Kaggle

Secrets live in Kaggle, never in this repo. Push the notebook once (step 4) so it exists in your account, then open it on Kaggle and go to Add-ons → Secrets. Add these four, with the names matching exactly, and make sure each one is attached to the notebook:

Secret name	What to put in it
NEWSDATA_API_KEY	Your API key from newsdata.io
GMAIL_ADDRESS	The Gmail address the brief is sent from
GMAIL_APP_PASSWORD	A Google app password for that sender account (not your normal password)
RECIPIENT_EMAIL	The address that receives the brief. It can be the same as the sender

How to get a Google app password

1. Open your Google Account and go to Security
2. Turn on 2-Step Verification if it is not already on
3. Open App passwords, create one, and copy the 16-character code
4. Paste that code into the GMAIL_APP_PASSWORD secret

Internet access must be on for the notebook (the NewsData API, RSS feeds, article scraping and email all need it). The "enable_internet": true line in kernel-metadata.json handles this when you push through the CLI.

3. Add your Kaggle credentials to GitHub

In your repository go to Settings → Secrets and variables → Actions and add KAGGLE_USERNAME and KAGGLE_KEY. Get the key from your Kaggle account settings by creating a new API token. The workflow uses these to push the notebook.

4. Run it once by hand

export KAGGLE_USERNAME="your_username"
export KAGGLE_KEY="your_api_key"

kaggle kernels push -p .

5. Set up the daily trigger

GitHub's built-in schedule is best-effort and can run hours late, so Sift AI is triggered by an external scheduler instead. Make sure the workflow file is on your default branch, because repository_dispatch only works from there.

a) Create a GitHub token for the scheduler

1. GitHub → Settings → Developer settings → Personal access tokens → Fine-grained tokens → Generate new token
2. Repository access: only select repositories → your sift-ai repo
3. Permissions → Contents: Read and write
4. Copy the token right away, since GitHub shows it only once. Never commit it or paste it anywhere public.

b) Test the token

read -s TOKEN    # paste the token, press Enter (nothing will show)

curl -i -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Accept: application/vnd.github+json" \
  https://api.github.com/repos/YOUR_GITHUB_USERNAME/sift-ai/dispatches \
  -d '{"event_type":"trigger_colab_run"}'

HTTP/2 204 means success, and a run appears in the Actions tab.

c) Schedule it on cron-job.org (free)

Setting	Value
Account timezone	Your own timezone (for example Asia/Kolkata)
URL	https://api.github.com/repos/YOUR_GITHUB_USERNAME/sift-ai/dispatches
Schedule	Every day at the time you want the brief to run
Method	POST
Headers	Authorization: Bearer YOUR_TOKEN, Accept: application/vnd.github+json, Content-Type: application/json
Body	{"event_type":"trigger_colab_run"}

Use Test run on the job and check for status 204.

Fine-grained tokens expire. Note the expiry date and renew the token in cron-job.org before it lapses, or the daily run will silently stop.

6. Let it run

From here the scheduler fires every day at the time you chose and a fresh brief arrives in your inbox. Allow some time after the trigger for Kaggle to queue and finish the notebook before you expect the email.

◈ Customize Your Feed

Open sift-ai.ipynb and edit the configuration cell near the top. Commit and push, and the next scheduled run uses your changes.

Run time. Change the time on the cron-job.org job, and nothing else is needed. The workflow also includes an optional guard step ("Check IST time window") that skips API-triggered runs outside a set hour range, so a late trigger never sends a stale brief. Edit the timezone and the two hour values in that step to match your schedule, or delete the step if you don't want a guard.

Topics you care about (USER_PREFERRED_TOPICS). These are a preference, not a filter. A great article outside your topics still beats a mediocre one inside them.

USER_PREFERRED_TOPICS = [
    "latest technology inventions",
    "politics",
    "artificial intelligence",
    "business",
    "technology",
    "economy",
    "world news",
    "AI companies",
]

Trusted sources (TRUSTED_SOURCES). Publications listed here get a credibility boost when the editor compares candidates. This is a soft signal, not a hard filter, so niche topics never end up with an empty pool. Names must be lowercase and match the publication name NewsData reports.

TRUSTED_SOURCES = {
    "reuters", "associated press", "ap", "bbc", "the guardian", "bloomberg",
    "the new york times", "the washington post", "npr", "the wall street journal",
    "the hindu", "the indian express", "hindustan times", "livemint",
    "business standard", "press trust of india", "pti",
}

Editorial and opinion feeds (EDITORIAL_FEEDS). These RSS feeds are read directly, because news APIs mostly index wire news rather than opinion sections. Add any publication that offers an RSS feed.

EDITORIAL_FEEDS = {
    "The Hindu": "https://www.thehindu.com/opinion/editorial/feeder/default.rss",
    "The Hindu (Op-Ed)": "https://www.thehindu.com/opinion/op-ed/feeder/default.rss",
    "Indian Express (Editorials)": "https://indianexpress.com/section/opinion/editorials/feed/",
    "Indian Express (Columns)": "https://indianexpress.com/section/opinion/columns/feed/",
    "Hindustan Times (Editorials)": "https://www.hindustantimes.com/feeds/rss/editorials/rssfeed.xml",
}

◈ Tech Stack

Layer	Tooling
Scheduling	cron-job.org (free) calling GitHub repository_dispatch
Orchestration	GitHub Actions
Compute	Kaggle GPU kernels · Nvidia T4 · Python 3.10
LLM inference	Ollama · Qwen 2.5 7B Instruct · Qwen 3 1.7B
Agents	CrewAI
Sources	NewsData API · RSS via feedparser
Scraping	requests · BeautifulSoup
Output	HTML template · WeasyPrint (PDF) · Gmail SMTP

◈ Design Decisions

Headless API, not browser automation

The first version drove Google Colab through Playwright. It worked until the DOM changed. Sift AI now talks to Kaggle directly through its official CLI.

	Legacy (Playwright + Colab)	Sift AI (Kaggle API)
Execution	Simulated browser clicks	Direct CLI dispatch
Authentication	Session cookies, UI logins	Scoped API key
Reliability	Breaks on DOM changes and captchas	Deterministic cloud execution
Overhead	Browser boot, DOM parsing	Lightweight API payload

External scheduler, not GitHub's schedule

GitHub's cron trigger runs on shared infrastructure and can start late, sometimes by hours. For a brief that has to land in the morning, that is too unreliable. A scheduler that calls the repository_dispatch API starts the workflow within seconds of the target time, and an optional time-window guard in the workflow prevents a stray late trigger from sending a stale brief.

Models judge, code handles the text

Scraping and cleaning are deterministic, so the article reaches you exactly as published. Models only do what they are good at: comparing candidates, extracting takeaways and teaching a word.

Open-weight models, not paid APIs

Qwen served through Ollama means no per-token billing and no vendor lock-in.

◈ Repository Structure

sift-ai/
├── .github/
│   └── workflows/
│       └── daily_colab_trigger.yml   # Dispatch trigger + optional time-window guard
├── assets/                           # README graphics
├── sift-ai.ipynb                     # Full pipeline: collect, select, summarize, PDF, email
├── kernel-metadata.json              # Kaggle GPU execution spec
└── README.md

◈ FAQ

Do I need a GPU or a server?

No. The GPU is a Kaggle T4 that spins up for the run and shuts down afterwards. A free external scheduler and GitHub Actions are the only things that wake it.

Can I trigger a run outside the schedule?

Yes. Use Actions → Run workflow (workflow_dispatch) at any time of day. API calls (repository_dispatch) run whenever they arrive, unless you keep the optional time-window guard in the workflow, in which case they are skipped outside the hours you set. You can also push the kernel yourself with the Kaggle CLI.

Why was my run skipped?

If you kept the optional "Check IST time window" step, it skips API-triggered runs outside the hours set in that step. Make sure those hours match the time on your cron-job.org job (and its timezone), edit or delete the step, or start the run manually from the Actions tab.

The daily run stopped working. What should I check?

Most often the GitHub token used by the scheduler has expired. Create a new one (Contents: Read and write), update it in cron-job.org, and use Test run to confirm a 204 response.

Can I send the brief to more than one person?

The notebook currently sends to the single address in RECIPIENT_EMAIL. To reach several people, extend the send step to loop over a list of addresses.

Can I use different models?

Yes. Any model Ollama can serve will work. Change the model names in the configuration cell, pull them in the setup cells, and make sure they fit in T4 memory.

Built by Manas

_{Sift through the noise. Keep the signal.}