How to split a large text file into smaller files with equal number of lines?

Question

I've got a large (by number of lines) plain text file that I'd like to split into smaller files, also by number of lines. So if my file has around 2M lines, I'd like to split it up into 10 files that contain 200k lines, or 100 files that contain 20k lines (plus one file with the remainder; being evenly divisible doesn't matter). I could do this fairly easily in Python but I'm wondering if there's any kind of ninja way to do this using bash and unix utils (as opposed to manually looping and counting / partitioning lines).

Answer 1

Choose your weapon:

split -l 200000 my_file

Then you'll need some script to clean up the resulting file names.

You can use sed to select a range of lines, but then you will need a bash loop to set the start and end files.

sed -n '$start,$endp' my_file > output.txt

A similar approach using awk:

cat my_file | awk -v Start=$start -v End=$end 'BEGIN{COUNT = 0}{if(COUNT >= Start && COUNT < End){print $0};COUNT=COUNT+1}' > output.txt