1.5 KiB
__ __ __
__/ // /_/ /___ _____ ____ _ __ ___ ____ __
/_ _ __/ / __ `/ __ \/ __ `/ / / / / | /| / / / / /
/_ _ __/ / /_/ / / / / /_/ / / /_/ /| |/ |/ / /_/ /
/_//_/ /_/\__,_/_/ /_/\__, / \__,_/ |__/|__/\__,_/
/____/
High level approach
We started by creating robust abstracted HTTP-handling code, which is located in the smol-http
module of this project. The HTTP code implements a subset of HTTP 1.1 which is enough to meet the
requirements for crawling the target web server. It also uses plain TCP sockets to communicate using
its HTTP implementation. We used Racket standard library functions to parse and manipulate URLs as
well as parse HTML (as XML, hopefully it's well-formed!) in order to find the hyperlinks on the page
as well as the flags. We implemented a high performance Certified Web Scale(tm) crawling scheduler
with a distributed work queue to allow for very high rate crawling, the crawler on our machines
takes minutes to complete, and finds all the flags very quickly.
Challenges
The current pandemic situation continues to make this semester difficult. Otherwise, we didn't run into any major issues during this project.
Testing
We unit tested the HTTP handling code in smol-http, and used ad-hoc manual testing against the target server to test the complete crawling functionality.
We have an additional -d flag which will print useful debug info during the execution of the
crawler, which may be helpful for manual testing.