A Workaround for CSV Upload Limits

Sometimes, applications have a limit on how many records you can import at once. Sometimes massive imports flag for a manual check, adding a roadblock to your work. Want to split one CSV into many quickly and avoid those pesky upload limits?

Here’s a short bash script I use to split .csv files into multiple files while maintaining the header rows:

<br />
#!/bin/bash</p>
<p>TARGET=$1</p>
<p>if [ ! -z $! ]; then<br />
    echo &quot;Filename is required&quot;<br />
    exit 1<br />
fi</p>
<p>COUNT=${2:-999}<br />
OUTPUT=$TARGET.split</p>
<p>mkdir $OUTPUT</p>
<p>tail -n +2 $TARGET | split -l $COUNT - split_</p>
<p>for file in split_*<br />
do<br />
    head -n 1 $TARGET &amp;gt; tmp_file<br />
    cat $file &amp;gt;&amp;gt; tmp_file<br />
    mv -f tmp_file $OUTPUT/$file.csv<br />
    rm $file<br />
done</p>
<p>echo &quot;all done!&quot;

Copy this into a file and run it from the terminal. I call it ‘niner’ on my machine. The first argument is the path to a CSV file. The second argument is how many rows you want in each segment. The default is 999, because scooting under a 1,000 row limit is why I wrote this in the first place 😉

Click Here to Leave a Comment Below 0 comments

Leave a Reply: