Artem Rys: RabbitMQ Scrapy Item Publisher in Python

It’s a small note about the Scrapy Item Pipeline that publishes a Scrapy Item to the RabbitMQ.

Photo by rawpixel on Unsplash

If you are interested in parsing/scraping projects in Python, you should know about Scrapy. Let’s imagine that your project is something bigger than a one-time data extraction. This means that you need to do something with your data. One of the possible approaches is to use RabbitMQ to publish items to some queue. This method can help you to build a custom consumer that can process Scrapy items.

Code (repository can be found out here):

https://medium.com/media/da4bfe94193999763fc0bfdfd3df38b2/href

Calling Celery Tasks not from Python

Thanks for the attention to the topic, feel free to leave your questions in the comments for discussion.

---

RabbitMQ Scrapy Item Publisher in Python was originally published in python4you on Medium, where people are continuing the conversation by highlighting and responding to this story.

from Planet Python
via read more