A Workaround for CSV Upload Limits

Sometimes, applications have a limit on how many records you can import at once. Sometimes massive imports flag for a manual check, adding a roadblock to your work. Want to split one CSV into many quickly and avoid those pesky upload limits?

Here’s a short bash script I use to split .csv files into multiple files while maintaining the header rows:

#!/bin/bash

TARGET=$1

if [ ! -z $! ]; then
    echo "Filename is required"
    exit 1
fi
    
COUNT=${2:-999}
OUTPUT=$TARGET.split

mkdir $OUTPUT

tail -n +2 $TARGET | split -l $COUNT - split_

for file in split_*
do 
    head -n 1 $TARGET > tmp_file
    cat $file >> tmp_file
    mv -f tmp_file $OUTPUT/$file.csv
    rm $file
done

echo "all done!"

Copy this into a file and run it from the terminal. I call it ‘niner’ on my machine. The first argument is the path to a CSV file. The second argument is how many rows you want in each segment. The default is 999, because scooting under a 1,000 row limit is why I wrote this in the first place 😉