How YouthAtlas Finds and Vets Opportunities Every Day

One of the most common questions we get is: "How do you find all these opportunities?" The answer is a custom-built AI pipeline that runs automatically every day at 4 AM UTC.

Here's how it works. We maintain a set of scrapers — automated programs that visit trusted opportunity aggregator websites daily. These aren't random sites; they're established platforms like YouthOp, Opportunities for Youth, OpportunityDesk, AfterSchool Africa, and ScholAds that have been curating opportunities for years. Our scrapers visit each source, identify new or updated listings, and extract the raw content.

Next comes extraction and validation. Each raw listing is processed by an AI model (Google Gemini 2.5 Flash) that extracts structured data: title, organization, description, eligibility, deadline, funding status, regions, opportunity type, and more. This structured data is then validated against a strict schema — if any required field is missing or malformed, the listing is flagged for human review rather than published with incorrect information.

Deduplication is critical. The same opportunity often appears on multiple source sites. Our pipeline uses a three-layer dedup system: URL matching (catches exact reposts), content hashing (catches identical text with different URLs), and fuzzy title matching (catches slight variations in how the same opportunity is named across sites).

Every validated opportunity also gets an embedding — a mathematical representation of its content generated by OpenAI's text-embedding model. These embeddings power our semantic search. When you search for "climate research funding," you'll find opportunities about environmental science, sustainability grants, and conservation fellowships — even if they don't contain the exact words you typed.

When we label something as "fully funded," it means tuition, housing, and a stipend are covered based on the listing's own description. We don't editorialize or inflate — what you see is what the source reported, structured and made searchable.

The result: 800+ vetted opportunities, updated every single day, searchable by meaning — not just keywords.