Node.js: remove specific columns from CSV file

I have a CSV file can contain around million records, how can I remove columns starting with _ and generate a resulting csv

For the sake of simplicity, consider i have the below csv

Sr.No Col1 Col2 _Col3   Col4 _Col5
1     txt  png  676766  win  8787
2     jpg  pdf  565657  lin  8787
3     pdf  jpg  786786  lin  9898

I would want the output to be


Sr.No Col1 Col2 Col4
1     txt  png  win 
2     jpg  pdf  lin 
3     pdf  jpg  lin

Do i need to read the entire file to achive this or is there a better approach to do this.

const csv = require('csv-parser');
const fs = require('fs');

fs.createReadStream('data.csv')
  .pipe(csv())
  .on('data', (row) => {
    // generate a new csv with removing specific column
  })
  .on('end', () => {
    console.log('CSV file successfully processed');
  });

Any help on how can i achieve this would be helpful.

Thanks.

3 thoughts on “Node.js: remove specific columns from CSV file”

  1. Actually you can handle that by using two npm packages.

    https://www.npmjs.com/package/csvtojson
    to convert your library to JSON format

    then use this
    https://www.npmjs.com/package/json2csv

    with the second library. If you know what are the exact fields you want. you can pass parameters to specifically select the fields you want.

    const { Parser } = require('json2csv');
     
    const fields = ['field1', 'field2', 'field3'];
    const opts = { fields };
     
    try {
      const parser = new Parser(opts);
      const csv = parser.parse(myData);
      console.log(csv);
    } catch (err) {
      console.error(err);
    }
    

    Or you can modify the JSON object manually to drop those columns

    Reply
  2. To anyone who stumbles on the post

    I was able to transform the csv’s using below code using fs and csv modules.

    await fs.createReadStream(m.path)
          .pipe(csv.parse({delimiter: '\t', columns: true}))
          .pipe(csv.transform((input) => {
            delete input['_Col3'];
            console.log(input);
            return input;
          }))
          .pipe(csv.stringify({header: true}))
          .pipe(fs.createWriteStream(transformedPath))
          .on('finish', () => {
            console.log('finish....');
          }).on('error', () => {
            console.log('error.....');
          });
    

    Source: https://gist.github.com/donmccurdy/6cbcd8cee74301f92b4400b376efda1d

    Reply
  3. Try this with csv lib

    const csv = require('csv');
    const fs = require('fs');
    
    const csvString=`col1,col2
                   value1,value2`
    
    csv.parse(csvString, {columns: true})
       .pipe(csv.transform(({col1,col2}) => ({col1}))) // remove col2
       .pipe(csv.stringify({header:true}))
       .pipe(fs.createWriteStream('./file.csv'))
    
    Reply

Leave a Comment