How to use scrapy to crawl a website which hides the url as href="javascript:;" in the next button

I am learning python and scrapy lately. I googled and searched around for a few days, but I don’t seem to find any instruction on how to crawl multiple pages on a website with hidden urls – <a href="javascript:;". Basically each page contains 20 listings, each time you click on “>>” button, it will load the next 20 items. I can’t figure out how to find the actual urls, below is the source code for your reference. Any pointers and help is greatly appreciated.

74 thoughts on “How to use scrapy to crawl a website which hides the url as href="javascript:;" in the next button”

  1. i need loan urgently in dubai, i need a direct loan lender. i need a loan have no credit need loan need a loan now i need a loan low interest, cash advance loans with a prepaid debit card, cash advance loans, cash advance loans, cash advance loans what do i need. Bank is typically viewed money management, terms of the payment .

    Reply

Leave a Comment