crawler
Concurrent web crawler built with C# and PostgreSQL.
Uses persistent frontier storage, atomic URL claiming, host-level politeness, and concurrent workers.
I'm a computer science student at Kent State. I'm interested in backend systems, search, networking, and building things from scratch.
Concurrent web crawler built with C# and PostgreSQL.
Uses persistent frontier storage, atomic URL claiming, host-level politeness, and concurrent workers.