How to get the content of a div tag when scraping with puppeteer and NodeJs

I heard of this library called puppeteer and is usefulness in scraping web pages. so i decided to scrape a gaming site content so I can store it data and go through it later.

But after i copied the XPATH of the div tag I want puppeteer to scrape it content, its returning Empty string Please what am I doing wrong.

This is the url am trying to scrape here

i want to scrape the div tag where the result of the 6 different color ball are being displayed.
so i can get the number of those colors every 45 seconds.

const puppeteer = require("puppeteer");

async function scrapeData(url){
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(url);

const [dataReceived] = await page.$x('/html/body/div[1]/div/div/div/footer/div[2]/div[1]/div/div[1]/div[2]/div/div');
const elContent = await dataReceived.getProperty('textContent');
const elValue = await elContent.jsonValue();
console.log("just testing");

here is the image of the returned empty string in vscode terminal

enter image description here

10 thoughts on “How to get the content of a div tag when scraping with puppeteer and NodeJs”

  1. Rather than using page.$x here, you could use a simpler selector, which would be less brittle. Try page.$('.ball-value'), or possibly page.waitForSelector('.ball-value') to deal with transition times. Testing on that page using a simpler selector seems to work. If you want to get all the ball values rather than just the first one, there’s page.$$ (which is the same as document.querySelectorAll, so it would return an array of elements).


Leave a Comment