How to merge arrays that are passed into the function?

So I’ve been working on my project that involves scraper.

So workflow is next: There are two scrapers right now. Data is being parsed and pushed into array for each individual scraper and passed to the merge component.

So the merge component looks like this:

let mergedApartments = []; //Creating merged list of apartments
exports.mergeData = (apartments) => {
  //Fetching all apartments that are passed from scraper(s)
  mergedApartments.push(...apartments); //Pushing apartments into the list
  console.log(mergedApartments.length);
};

So right now output of mergedApartments.length is 9 39. So the first function that calls mergeData() and pass it an array have 9 objects inside it, and the other scraper have 30 objects inside it’s array, who is again passed to the mergeData.

Now this is not what I’ve expected. I’ve expected one array with all merged objects from the scrapers. Right now, scraperno1 send apartments and it’s added to the mergedApartments, then scraperno2 sends apartments and it’s overwriting that array by adding new apartments objects into the array.

Now I want different output: I just want to get one list with all merged objects from the arrays. Because this data will be passed to the storing component, and I don’t want to query DB multiple times, because for each new mergedApartments list, data will be inserted and creating duplicate values – throwing an error.

So what I’ve tried: I’ve tried creating some kind of a counter which counts number of time that function mergeData is called, and then do the logic about merging but no success.

So I just want my array to have one output of mergedApartments.length – in this case 39.

Thanks!

EDIT
Here how one of the scraper looks:

const merge = require('../data-functions/mergeData');
const axios = require('axios');
const cheerio = require('cheerio');

//function for olx.ba scraper. Fetching raw html data and pushing it into array of objects. Passing data to merge function

exports.santScraper = (count) => {
  const url = `https://www.sant.ba/pretraga/prodaja-1/tip-2/cijena_min-20000/stranica-${count}`;

  const santScrapedData = [];
  const getRawData = async () => {
    try {
      await axios.get(url).then((response) => {
        const $ = cheerio.load(response.data);

        $('div[class="col-xxs-12 col-xss-6 col-xs-6 col-sm-6 col-lg-4"]').each(
          (index, element) => {
            const getLink = $(element).find('a[class="re-image"]').attr('href');
            const getDescription = $(element).find('a[class="title"]').text();
            const getPrice = $(element)
              .find('div[class="prices"] > h3[class="price"]')
              .text()
              .replace(/\.| ?KM$/g, '')
              .replace(',', '.');
            const getPicture = $(element).find('img').attr('data-original');
            const getSquaremeters = $(element)
              .find('span[class="infoCount"]')
              .first()
              .text()
              .replace(',', '.')
              .split('m')[0];
            const pricepersquaremeter =
              parseFloat(getPrice) / parseFloat(getSquaremeters);

            santScrapedData[index] = {
              id: getLink.substring(42, 46),
              link: getLink,
              descr: getDescription,
              price: Math.round(getPrice),
              pictures: getPicture,
              sqm: Math.round(getSquaremeters),
              ppm2: Math.round(pricepersquaremeter),
            };
          }
        );
        merge.mergeData(santScrapedData); //here i'm calling function and passing array to function
      });
    } catch (error) {
      console.log(error);
    }
  };
  getRawData();
};

Other scraper looks the same(it’s same calling of the function)

36 thoughts on “How to merge arrays that are passed into the function?”

  1. 601619 722544I just could not go away your site before suggesting that I really enjoyed the usual information a person supply to your guests? Is going to be back ceaselessly to be able to inspect new posts. 34073

    Reply
  2. 912910 2303You can definitely see your skills in the work you write. The world hopes for a lot more passionate writers like you who arent afraid to say how they believe. At all times follow your heart 199294

    Reply

Leave a Comment