PSF GSoC students blogs: [Blog #5] Time just seems to fly.

 

<meta charset="utf-8">

Hello! This is my second last blog post for GSoC 2019 - time has gone by so quickly. I spend this week documenting Protego’s API in detail. I opened a pull request to add Protego integration in Scrapy. I added PyPy test environment and modified Protego to treat non-terminal dollar sign as ordinary character.

Up next, I will start the process to transfer Protego to Scrapy organisation on GitHub. I would modify `SitemapCrawler` in Scrapy to use the new interface, and implement a `ROBOTSTXT_USER_AGENT` setting in Scrapy.

I faced minor problems trying to setup PyPy environment in Travis. With the help from mentors, I was able to resolve the issue.

from Planet Python
via read more