Puppeteer timeout on specific website running on server cloud headless

I’ve made a node.js web scraper code that runs fine on my computer, however, when I deploy to my Google Cloud VM instance running Debian, it returns a timeout error for a specific website. I’ve tried many differnent setups for puppeteer, but none seems to work. I believe the website I’m trying to scrape is blocking my code when I run from the google cloud server, but not when I run from my computer. The scraping part is working fine on my computer. Puppeteer finds the HTML tags and retrieve the info.

const puppeteer = require('puppeteer');
const GoogleSpreadsheet = require('google-spreadsheet');
const { promisify } = require('util');
const credentials = require('./credentials.json');

async function main(){

    const scrapCopasa = await scrapCopasaFunction();

    console.log('Done!')

}



async function scrapCopasaFunction() {

    const browser = await puppeteer.launch({
        args: ['--no-sandbox'], 
    });
    const page = await browser.newPage();
    //await page.setDefaultNavigationTimeout(0);
    //await page.setViewport({width: 1366, height: 768});
    await page.setUserAgent('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36');
    await page.goto('http://www.copasa.com.br/wps/portal/internet/abastecimento-de-agua/nivel-dos-reservatorios');
    //await new Promise(resolve => setTimeout(resolve, 5000));
    
    let isUsernameNotFound = await page.evaluate(() => {
        if(document.getElementsByClassName('h2')[0]) {
            if(document.getElementsByTagName('h2')[0].textContent == "Sorry, this page isn't available.") {
                return true;
            }
        }
    });

    if(isUsernameNotFound) {
        console.log('Account not exists!');        
        await browser.close();
        return;
    }


    let reservoirLevelsCopasa = await page.evaluate(() => {
        const tds = Array.from(document.querySelectorAll('table tr td'))
        return tds.map(td => td.innerText)        
    });


    const riomanso = reservoirLevelsCopasa[13].replace(",",".").substring(0,5);
    const serraazul = reservoirLevelsCopasa[17].replace(",",".").substring(0,5);
    const vargemdasflores = reservoirLevelsCopasa[21].replace(",",".").substring(0,5);

    await browser.close();

    return[riomanso, serraazul, vargemdasflores];

}



main();

And error that I’m getting is the following:

(node:6425) UnhandledPromiseRejectionWarning: TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded
    at /home/xxx/reservoirs/node_modules/puppeteer/lib/LifecycleWatcher.js:142:21
    at async FrameManager.navigateFrame (/home/xxx/reservoirs/node_modules/puppeteer/lib/FrameManager.js:94:17)
    at async Frame.goto (/home/xxx/reservoirs/node_modules/puppeteer/lib/FrameManager.js:406:12)
    at async Page.goto (/home/xxx/reservoirs/node_modules/puppeteer/lib/Page.js:674:12)
    at async scrapCopasaFunction (/home/xxx/reservoirs/reservatorios.js:129:5)
    at async main (/home/xxx/reservoirs/reservatorios.js:9:25)
  -- ASYNC --
    at Frame.<anonymous> (/home/xxx/reservoirs/node_modules/puppeteer/lib/helper.js:111:15)
    at Page.goto (/home/xxx/reservoirs/node_modules/puppeteer/lib/Page.js:674:49)
    at Page.<anonymous> (/home/xxx/reservoirs/node_modules/puppeteer/lib/helper.js:112:23)
    at scrapCopasaFunction (/home/xxx/reservoirs/reservatorios.js:129:16)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async main (/home/xxx/reservoirs/reservatorios.js:9:25)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:6425) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async f
unction without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled
 promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode)
. (rejection id: 1)
(node:6425) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not ha
ndled will terminate the Node.js process with a non-zero exit code.

104 thoughts on “Puppeteer timeout on specific website running on server cloud headless”

  1. Pingback: viagra sale uk
  2. Pingback: aralen 250
  3. Pingback: lumigan 01
  4. Pingback: molnupiravir
  5. Pingback: tamoxifen 5 mg
  6. Pingback: olumiant price
  7. Pingback: 12 mg zanaflex
  8. Greetings! I know this is kind ᧐f off topic Ƅut I waѕ
    wondering which blog platform ɑre y᧐u using fоr this website?
    I’m getting sick аnd tired of WordPress Ьecause
    І’ve һad pгoblems wіth hackers and I’m ⅼooking at options fоr аnother platform.
    I would be awesome if yօu couⅼɗ point me in the direction of
    a good platform.

    Reply
  9. com 20 E2 AD 90 20Viagra 20100 20Mg 20Kullanm 20Ekli 20 20Kamagra 20Viagra 20Kulonbseg viagra 100 mg kullanm ekli Milliner was unable to participate in offseason workouts while he recovered from shoulder surgery, missed the beginning of training camp due to a short holdout, was called out by Ryan for playing poorly in the teamГў s second preseason game, suffered an Achilles injury, was benched against New England in Week 2 and now has the issue with his left hamstring cialis generic buy economy will give the Fedroom to begin scaling back its bond buying, most likely inSeptember, have sparked a near 5 percent rally in the dollar andsome 50 basis point rise in the benchmark 10 year U

    Reply
  10. The analysis suggests that, for ospemifene indicated in postmenopausal women with VVA who are not candidates for local estrogens, similar efficacy to that observed with local vaginal estrogens can be expected, with a comparable, but slightly different safety profile cialis for daily use Ohtani K, Ide T, Hiasa KI, et al

    Reply

Leave a Comment