What did I do this week?
I have finished my work on improving cvedb this week. I am using aiohttp to download NVD dataset instead of requesting with multiprocessing pool. This has improved our downloading speed since now every tasks are downloading concurrently in same thread instead of 4 tasks at a time with process pool. I have also measured performance of aiosqlite but it was significantly slower while writing to database so, I decided to keep writing process synchronous. I have also added a beautiful progressbar with the help of rich module. So, now user can get feedback about progress of the downloading and updating database. Here is the demo of how does it look now.
It was taking 2 minutes to download and update database with multiprocessing. Now, it is only taking 1 minute for that. So, we have gained 200% speed just by converting IO bound tasks into asynchronous coroutines. I have also fixed an event loop bug that we are getting sometimes due to parallel execution of pytest. I have also fixed a small bug in extractor in PR #767. I have also created some utils function to reduce code repetition.
What am I doing this week?
I have started working on input engine and I am hoping to provide basic triage support by the end of this week.
Have I got stuck anywhere?
No, I didn't get stuck anywhere this week :)
from Planet Python
via read more
No comments:
Post a Comment