Building Automated Curated Lists with GitHub and Packagist

The refresh.js script iterates entries, calls https://repo.packagist.org/p2/{vendor}/{package}.json and https://api.github.com/repos/{owner}/{repo}, then applies:

• archived — gh.archived === true (author declared done)
• abandoned — packagist.abandoned === true (ecosystem explicit field)
• stale — last_push > 365 days ago (inference, warns only)

Archived and abandoned entries get status: "removed" and are excluded from the rendered list. Stale entries keep status: "stale" so a human can decide. The rendered table shows live stars, installs, and the refresh timestamp so readers know the data is current or visibly not.

Data schema and validation gate

Every entry lives in data/packages.json with this shape:

{
 "id": "vendor/package",
 "category": "testing",
 "summary": "One line, no period, max 120 chars",
 "github": "owner/repo",
 "packagist": "vendor/package",
 "status": "active",
 "stars": 1234,
 "installs": 56789,
 "last_push": "2024-11-12T14:30:00Z",
 "last_refreshed": "2025-01-15T06:17:00Z"
}

A pre-flight validation script runs before any API call. It catches duplicates, unknown categories, malformed vendor/package names, and missing summaries in ~2 seconds. This saves hundreds of network round trips on bad PRs.

const CATEGORIES = new Set([
 'testing','static-analysis','async','cli','http','database',
 'templating','security','utilities','dev-tools'
]);

function validate(entries) {
 const seen = new Set();
 for (const [idx, pkg] of entries.entries()) {
 const id = `${pkg.vendor}/${pkg.package}`;
 if (seen.has(id)) throw new Error(`duplicate entry: ${id}`);
 if (!CATEGORIES.has(pkg.category)) throw new Error(`${id} points at unknown category`);
 if (!pkg.summary?.trim()) throw new Error(`${id} is missing "summary"`);
 if (!/^[a-z0-9-]+\/[a-z0-9-.]+$/i.test(id)) throw new Error(`${id} malformed vendor/package`);
 seen.add(id);
 }
}

A contributor who typos a category gets feedback in three seconds instead of after a failed API batch.

What broke: GitHub API rate limits on the first run

The initial research list had 63 packages. The first scheduled run hit GitHub's unauthenticated limit (60 req/hr) because I forgot to pass the GITHUB_TOKEN to the octokit client in the script. The workflow failed at entry 61 with 403 rate limit exceeded. Fix: instantiate Octokit with auth: process.env.GITHUB_TOKEN and enable throttle plugin. Second run completed in 47 seconds for 58 packages. Lesson: always authenticate even for public repos; the authenticated limit is 5,000 req/hr and costs nothing.

Pre-publish hygiene: run the refresh before the first commit

I ran the refresh script locally against my initial 63-package research list before publishing. Five entries were already archived or abandoned on Packagist — they looked fine in search results but were dead. Publishing first and automating later would have shipped all five in the launch announcement. The research phase is when your data is most stale because search engines have no opinion on project liveness. Run the freshness check as the final step before git push.

Rendering the list with a timestamp readers can trust

The render script (scripts/render.js) reads the refreshed JSON, groups by category, and writes a Markdown table per category. Each table header includes Last refreshed: 2025-01-15 06:17 UTC. Stars and installs come from the API, not hand-typed numbers. A hand-typed star count is wrong the day after you type it; a reader has no way to tell how stale it is. The timestamp makes the freshness visible.

function renderTable(entries, category) {
 const lines = [
 `## ${category}`,
 '',
 '| Package | Summary | Stars | Installs | Status |',
 '|---------|---------|-------|----------|--------|'
 ];
 for (const e of entries) {
 const badge = e.status === 'stale' ? ' stale' : 'active';
 lines.push(`| [${e.id}](${e.github_url}) | ${e.summary} | ${e.stars} | ${e.installs} | ${badge} |`);
 }
 lines.push('', `*Last refreshed: ${new Date().toISOString().replace('T',' ').replace('Z',' UTC')}*`);
 return lines.join('\n');
}

When NOT to use this method

• Tiny lists (<15 entries) — manual audit twice a year is faster than maintaining the workflow.
• Non-programmatic ecosystems — if the target platform has no API (some design asset sites, course directories), you cannot automate freshness.
• Curated opinion over signal — if the list's value is "I personally vetted these," automated removal removes your judgment. This method assumes the signal (archived, abandoned, idle) is authoritative.
• One-off re — a conference talk re

Comparison: automated freshness vs. static awesome list

• maintenance burden — automated: ~30 min/month (review stale flags); static: ~4 hrs/quarter (full re-audit) or rots
• reader trust — automated: live numbers + timestamp; static: unknown age, often years stale
• false positives — automated: stale flag on finished-but-useful libs (mitigated by warn-don't-remove); static: none, but false negatives dominate
• setup cost — automated: 4 hrs initial; static: 0
• ecosystem fit — automated: needs package registry API + GitHub; static: works anywhere

Extending to other ecosystems

The pattern ports to any language with a package registry API. For npm, replace Packagist calls with https://registry.npmjs.org/{package} and check deprecated field. For PyPI, use https://pypi.org/pypi/{package}/json and watch for yanked releases. The GitHub archived/stale logic stays identical. The validation gate, render script, and workflow skeleton are reusable; only the fetcher module changes.

Minimal

1. Create data/packages.json with 10 entries, the schema above, and a categories array.
2. Add .github/workflows/refresh.yml with the schedule and Node steps.
3. Write a 40-line scripts/refresh.js that fetches stars and pushed_at from GitHub only (skip Packagist if not PHP).
4. Write scripts/render.js that outputs a single Markdown file.
5. Run locally: node scripts/refresh.js && node scripts/render.js. Commit the updated JSON and README.
6. Push. The first scheduled run will either pass or teach you what's missing.

Total setup: ~2 hours if you copy the fetcher pattern. The validation gate can wait until you accept external PRs.

What the list looks like after six months

At month six, the list shows 54 active entries, 3 flagged stale (all small finished utilities), and 1 removed (abandoned on Packagist). The README timestamp reads Last refreshed: 2025-06-15 06:17 UTC. Stars and installs on every row reflect that morning's API response. No manual edits since launch. The three stale entries are legitimate "finished" packages — a human will likely keep them. The automation didn't guess; it surfaced the decision.