How do I use cheerio to scrape dynamic websites?

First of all, please understand that grammar may not be correct by using a translator.

I’m going to use cheerio to do web scraping in React environment.

Part of the site(for example) :

<ul>
  <li>
    <div class="name">burger</div>
    <div class="price">5,500</div>
  </li>
  <li>
    <div class="name">sandwich</div>
    <div class="price">3,500</div>
  </li>
  <li>
    <div class="name">ramyeon</div>
    <div class="price">1,500</div>
  </li>
</ul>

my code(FYI, this code works well when scrap other sites) :

const cheerio = require("cheerio");
let prodData = [];

useEffect(() => {
    scraping();
}, []);

const scraping = () => { 

     axios.get("/product/thisIsExample")
        .then(res => {
            if (res.status === 200) {
                const html = res.data;
                const $ = cheerio.load(html);
                const children = [...$("ul").children("li")];
                children.forEach(v => {
                    prodData.push({
                        prodName: $(v).find("div.name").text(),
                        prodPrice: $(v).find("div.price").text()
                    });

                });
                
                if(prodData.length !== 0) {
                    console.log(prodData);
                }
            }
        }, (err) => console.log("error"));

}

The problem is that the part(<li>) I’m trying to scrape is dynamic, so generated only after the data call is finished.

I mean, I’m trying to scrape the <li> from <ul> into an array, but when I scrape it, there are no <li> inside <ul>.

What should I do to scrape <li>?

141 thoughts on “How do I use cheerio to scrape dynamic websites?”

  1. Pingback: stromectol pills
  2. Hi there! I realize this is sort of off-topic however I
    needed to ask. Does building a well-established blog
    like yours require a massive amount work? I’m completely new to operating a blog however I do write in my diary every day.
    I’d like to start a blog so I can easily share my own experience
    and feelings online. Please let me know if you have any kind of recommendations or tips for brand new aspiring bloggers.

    Appreciate it! http://harmonyhomesltd.com/Ivermectinum-tablets.html

    Reply

Leave a Comment