Removing a blank line from a text file

Posted by: tahir

Removing a blank line from a text file - 29/04/2005 15:54

How do I do this? It's on Linux, a friend suggested:

cat file | sed “/^.$/d” > newfile

but I've still got a LF at the end of the file (I think).
Posted by: wfaulk

Re: Removing a blank line from a text file - 29/04/2005 17:28

Most Unix text editors will put a LF at the end of the last line, sed probably included. You'll probably need a binary editor.

If you have it already, and you might, use vim. Load the file into vim with the -b option (vim -b filename), then, inside vim, unset the endofline option (:set noendofline), then save (:wq).

You can compare by running od on the file before and after. A good way would be "tail -1 filename | od -xc".
Posted by: Mataglap

Re: Removing a blank line from a text file - 29/04/2005 19:55

Code:
# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
 sed 's/.$//'               # assumes that all lines end with CR/LF
 sed 's/^M$//'              # in bash/tcsh, press Ctrl-V then Ctrl-M
 sed 's/\x0D$//'            # gsed 3.02.80, but top script is easier

 # IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
 sed "s/$/`echo -e \\\r`/"            # command line under ksh
 sed 's/$'"/`echo \\\r`/"             # command line under bash
 sed "s/$/`echo \\\r`/"               # command line under zsh
 sed 's/$/\r/'                        # gsed 3.02.80

http://www.student.northpark.edu/pemente/sed/sed1line.txt

http://www.google.com/search?q=sed+dos+unix
Posted by: wfaulk

Re: Removing a blank line from a text file - 29/04/2005 20:46

So far, you've got three completely different answers. Let's figure out what it is you're trying to do.

The file you attached has no blank lines, and its last line ends with a newline, as is customary with a Unix text file, of which CSVs are a specific kind.

I assume you don't want to delete blank lines, as your sample contains none. If you do, that's what the command you stated will do, almost. It'll actually delete every line that contains only a single character.

Mataglap's answers will convert DOS/Windows text files to Unix text files and the other way around. (Although the nearly ubiquitous dos2unix and unix2dos commands will do the same thing.) That doesn't seem to match your description, though.

If you want to delete the final newline from a text file, what I suggested will work, although there are other ways, few of them one-liners, since Unix command line utilities are designed to work with text files and what you want to do would make it a non-standard text file.

On the other hand, maybe you're confused about the trailing newline and you think it's not supposed to be there. It is. Are you having a problem with the CSV file that makes it look like you have an extra blank line?
Posted by: gbeer

Re: Removing a blank line from a text file - 30/04/2005 01:43

When that file is opened with Wordpad, the insertion point can be moved to a blank line after the last line of data. A backspace will move the insertion point to the end of the last line of data.

I suspect he wishes to concatenate two such files together in a gapless form so there are no blank lines in the spreadsheet.
Posted by: peter

Re: Removing a blank line from a text file - 30/04/2005 17:45

Quote:
I suspect he wishes to concatenate two such files together in a gapless form so there are no blank lines in the spreadsheet.

But the file he posted is already in that form. If the final LF were deleted, concatenating two files would result in the first line of the second appearing alongside (concatenated to) the last line of the first, not after it as desired.

Peter
Posted by: Mataglap

Re: Removing a blank line from a text file - 30/04/2005 21:14

And if it's the EOF marker that's the issue under discussion, it really won't matter. If it's cat'ed with another file the OS will take care of it properly. In the unlikely case that it doesn't happen it's still a text file, so there will just be a weird line in the middle of the file that's easy to find or fix.

I don't understand what the issue is other than the classic DOS vs. UNIX EOL issue, which can easily be handled with any of the solutions above, tools like dos2unix/unix2dos or flip which are likely either installed or easily installed for any Linux distro. (If we're talking about another Unix os like IRIX or HP-UX or something sed might be the easiest choice.)

By hand it's just a matter of adding or removing a "^M" / "LF" / "\r" / "\d013" / "\o015" / "\00d" (pick your method of entering a representation of the character) from the end of each line, easily done with sed on Unix or on Windows with cygwin. Looking for "." is bad because that means "any single character" which will strip off the last character from the line whether it's a ^M or not, so just anchor (with the "$") a ^M at the end of the line.

I know of at least one free win32 text editor (PFE) that can easily save the file in either format via the bottom status bar indicator, and I'd really be surprised if some of the others like TextPad &c. can't do the same as well.

--Nathan
Posted by: tahir

Re: Removing a blank line from a text file - 01/05/2005 05:10

Quote:
If you want to delete the final newline from a text file, what I suggested will work

On the other hand, maybe you're confused about the trailing newline and you think it's not supposed to be there. It is. Are you having a problem with the CSV file that makes it look like you have an extra blank line?


The file is submitted via an upload process and the last LF makes it unreadable for those purposes, the program that reads in the uploaded file can't deal with an extra LF at the end of the file (HM Customs & Excise)

Thanks so far
Posted by: tahir

Re: Removing a blank line from a text file - 01/05/2005 05:23

Quote:
If you have it already, and you might, use vim. Load the file into vim with the -b option (vim -b filename), then, inside vim, unset the endofline option (:set noendofline), then save (:wq).


Can this be automated by a script file? The rest of the process is automatic and I need this to be as well if possible.
Posted by: tahir

Re: Removing a blank line from a text file - 01/05/2005 05:42

BTW just so you know vim, cat, vi etc scare the crap out of me
Posted by: wfaulk

Re: Removing a blank line from a text file - 01/05/2005 05:48

If that's really what you want to do, and, again, it's odd, and you want to automate it, I'd say a better way would be to write a program to do it. If you don't feel up to that, a fairly unoptimized one-liner might be:

Code:
dd if=file of=newfile bs=1 count=`expr \`wc -c file | awk '{print $1}'\` - 1`

Ugh. Ugly. But it works.
Posted by: tahir

Re: Removing a blank line from a text file - 01/05/2005 05:52

Many thanks Bitt

Knowing very little about file manipulation under Linux I'm going to try and attack this from the other end, I'll see if I can change the way the file is generated to remove the last LF.
Posted by: wfaulk

Re: Removing a blank line from a text file - 01/05/2005 06:10

Well, like I said, that command will work. A more optimized one might be:

Code:
perl -pe 'BEGIN { $RS = undef; } chomp;' < file > newfile

or even:

Code:
perl -pie 'BEGIN { $RS = undef; } chomp;' file

(that one will overwrite the file in-place instead of having to start working with a new file as well as using built-in perl IO instead of relying on file descriptor redirection)

Anyway, they should be faster, assuming you have perl installed (you probably do) and assuming perl's read/write code is better optimized than the 1-byte-at-a-time dd-based one-liner I suggested above.
Posted by: ricin

Re: Removing a blank line from a text file - 01/05/2005 06:39

Mmmmm PIE!
Posted by: tahir

Re: Removing a blank line from a text file - 01/05/2005 09:05

Many thanks again for all your comments, I hadn't realised but the application that generated the .csv was adding CR/LF as it's default EOR delimiter, once I worked out how to change it to CR it loaded fine.

One of these days I should really do a crash course in linux.
Posted by: bonzi

Re: Removing a blank line from a text file - 01/05/2005 15:48

Quote:
"^M" / "LF" / "\r" / "\d013" / "\o015" / "\00d"

<nitpicking>
"^M" / "CR" / "\r" / "\d013" / "\o015" / "\x00d"
"^J" / "LF" / "\n" / "\d010" / "\o012" / "\x00a"
</nitpicking>

FWIW, my favourite free win32 editor is gvim (beware, www.gvim.org leads to a search site that tries to sneakily download and install some rubish!). It can save text files with DOS, Unix or MacOS EOLs, too.
Posted by: mlord

Re: Removing a blank line from a text file - 01/05/2005 15:56

Quote:
sneakily download and install some rubish


What browser would permit that???
Posted by: bonzi

Re: Removing a blank line from a text file - 01/05/2005 16:19

Quote:
What browser would permit that???

I guess this is a rhetorical question, and yes, I am guilty of running IE (actually, a wrapper around it called Crazy Browser - I slightly prefer its tabbed browser windows manipulation to that on FireFox). Of course, running IE I also have to run all kinds of anti-this and anty-that band-aid fxes to gaping built-in security holes.

I could also finally overcome my laziness, stop using work laptop from home (which, because of my dear clients, has to have windows on it) and configure myself a decent Linux machine...
Posted by: mlord

Re: Removing a blank line from a text file - 01/05/2005 16:22

Quote:
configure myself a decent Linux machine...


Heck, no -- use OSX if it makes you happy! Linux isn't the only decent system out there!

Cheers
Posted by: bonzi

Re: Removing a blank line from a text file - 01/05/2005 16:33

Quote:
Heck, no -- use OSX if it makes you happy! Linux isn't the only decent system out there!

I am tempted, actually. Perhaps a Mini in the living room... But really cool machines like those with dual G5 are still way more expensive than x86 based stuff, at least here.

I think it is going to be a dual-boot x86 (AMD, I guess) - Windows for an occasional game (strange, Microsoft seems to be more successful in making a solid product with Flight Simulator than with their OSes ) and Linux for everything else (now that support for Latin2 has matured).
Posted by: canuckInOR

Re: Removing a blank line from a text file - 02/05/2005 06:20

Quote:
Well, like I said, that command will work. A more optimized one might be:
Code:
perl -pe 'BEGIN { $RS = undef; } chomp;' < file > newfile
or even:
Code:
perl -pie 'BEGIN { $RS = undef; } chomp;' file


IIRC, chomp() removes $RS, and, since you've set that to undef to slurp the whole file, chomp is a no-op -- see 'perldoc -f chomp' -- so I'm guessing this wouldn't modify the file at all. Not to mention, in the -pie version, the -i flag takes a parameter, which sucks up the 'e' flag. You need to use "perl -pi -e", instead. On the subject of flags, you'll also want "-MEnglish" -- $RS is in the English module.

Code:
perl -e'{local $/=undef; @a=<>;}chomp $a[-1];print @a' < file > newfile

or
Code:
perl -pi -e'BEGIN{$/=undef;}s/\n$//' file

One caveat to epwc -- both of these slurp the entire file into memory, so if the attached example file was only a snippet of a very large file, you may not want to use this method.
Posted by: wfaulk

Re: Removing a blank line from a text file - 02/05/2005 12:12

You're right. I tested the first one and then assumed the second would work. And I may have typoed chomp for chop in both cases.