Thread Subject: Textread

I have a text file with the following data:

72008,16193 -0,01 0,02 0,015
72008,16375 -0,029 -0,034 -0,034
72008,16406 -0,02 -0,054 -0,059
72008,16437 -0,024 -0,039 -0,039
72008,16467 0,02 -0,044 -0,044
72008,16498 -0,005 -0,005 -0,005
72008,16529 0,034 0,015 0,01
72008,1656 0,01 0,034 0,034
72008,16591 0,029 0,01 0,01

I want to load the data in Matlab and replace the comma with dots. Can I use the textread function and load it into an cell array and replace the commas with dots or how do I do it?

I would be very happen if somebody could help me:)

On 1 Jul, 11:41, "Jesper Lauridsen" <jesperhols...@hotmail.com> wrote:
> I have a text file with the following data:
>
> 72008,16193     -0,01           0,02            0,015
> 72008,16375     -0,029          -0,034          -0,034
> 72008,16406     -0,02           -0,054          -0,059
> 72008,16437     -0,024          -0,039          -0,039
> 72008,16467     0,02            -0,044          -0,044
> 72008,16498     -0,005          -0,005          -0,005
> 72008,16529     0,034           0,015           0,01
> 72008,1656      0,01            0,034           0,034
> 72008,16591     0,029           0,01            0,01
>
> I want to load the data in Matlab and replace the comma with dots. Can I use the textread function and load it into an cell array and replace the commas with dots or how do I do it?
>
> I would be very happen if somebody could help me:)

This one is straight-forward, but might take some
time to run:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
f1id = fopen('myfile.txt');
f2id = fopen('repairedfile.txt','w');

while (~feof(f1id))
   s = getl(f1id);
   sidx = find(s==',');
   s(sidx) = '.';
   fprintf(f2id,'%s\n',s);
end

fclose(f1id);
fclose(f2id)'
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Rune

"Jesper Lauridsen" <jesperholst_5@hotmail.com> wrote in message <h2favd$6b6$1@fred.mathworks.com>...
> I have a text file with the following data:
>
> 72008,16193 -0,01 0,02 0,015
> 72008,16375 -0,029 -0,034 -0,034
> 72008,16406 -0,02 -0,054 -0,059
> 72008,16437 -0,024 -0,039 -0,039
> 72008,16467 0,02 -0,044 -0,044
> 72008,16498 -0,005 -0,005 -0,005
> 72008,16529 0,034 0,015 0,01
> 72008,1656 0,01 0,034 0,034
> 72008,16591 0,029 0,01 0,01
>
> I want to load the data in Matlab and replace the comma with dots. Can I use the textread function and load it into an cell array and replace the commas with dots or how do I do it?
>
> I would be very happen if somebody could help me:)

Does it have to be done in Matlab? Couldn't you open it in a text editor and use the 'find and replace tool'? Otherwise I think this code does what you want:
data = importdata('data.txt');
dlmwrite('data_out.txt',data,'.')

On 1 Jul, 12:40, "Samuel Dorrell" <sdorr...@asl-vision.co.uk> wrote:
> "Jesper Lauridsen" <jesperhols...@hotmail.com> wrote in message <h2favd$6b...@fred.mathworks.com>...
> > I have a text file with the following data:
>
> > 72008,16193        -0,01           0,02            0,015
> > 72008,16375        -0,029          -0,034          -0,034
> > 72008,16406        -0,02           -0,054          -0,059
> > 72008,16437        -0,024          -0,039          -0,039
> > 72008,16467        0,02            -0,044          -0,044
> > 72008,16498        -0,005          -0,005          -0,005
> > 72008,16529        0,034           0,015           0,01
> > 72008,1656         0,01            0,034           0,034
> > 72008,16591        0,029           0,01            0,01
>
> > I want to load the data in Matlab and replace the comma with dots. Can I use the textread function and load it into an cell array and replace the commas with dots or how do I do it?
>
> > I would be very happen if somebody could help me:)
>
> Does it have to be done in Matlab? Couldn't you open it in a text editor and use the 'find and replace tool'?

One argument for using matlab would be that the job has
to be repeated for several files.

These kinds of things often occur because of the locale
settings on some computer. On one occasion we found this
very flaw after having delivered hundreds of files to
a paying client. This was a 24/7 operation, and the flaw
was discovered just after I had finished my shift. Next
morning, one guy at the nigh shift had spent 12 hrs
doing the substitution manually:

- Open the file in an editor
- Do the find-replace thing
- Save the file

The only problem was that he had done it in an internal
database, while it was up to me to hand the corrected
files to the clients. For bueraucratic reasons, I had
to either wrap all the newly correcte files (300 if I
remember correctly) nicley in red tape, or repeat the
corrections in my archive of files that were already
'buearaucratically correct' but which still contained
the flaw.

So I wrote a matlab script to do the job. It was slow
compared to the C++ stuff I would have preferred to use,
but matlab was all I had. I got the job - start writing
the script to finished conversion - done in less than
2 hrs.

Very convenient, when you have a bunch of P-O'd clients
and bosses hanging around, fuming with rage and
frustrations.

Rune

On Wed, 01 Jul 2009 07:00:31 -0400, Rune Allnor <allnor@tele.ntnu.no>
wrote:

> On 1 Jul, 12:40, "Samuel Dorrell" <sdorr...@asl-vision.co.uk> wrote:
>> "Jesper Lauridsen" <jesperhols...@hotmail.com> wrote in message
>> <h2favd$6b...@fred.mathworks.com>...
>> > I have a text file with the following data:
>>
>> > 72008,16193        -0,01           0,02            0,015
>> > 72008,16375        -0,029          -0,034          -0,034
>> > 72008,16406        -0,02           -0,054          -0,059
>> > 72008,16437        -0,024          -0,039          -0,039
>> > 72008,16467        0,02            -0,044          -0,044
>> > 72008,16498        -0,005          -0,005          -0,005
>> > 72008,16529        0,034           0,015           0,01
>> > 72008,1656         0,01            0,034           0,034
>> > 72008,16591        0,029           0,01            0,01
>>
>> > I want to load the data in Matlab and replace the comma with dots.
>> Can I use the textread function and load it into an cell array and
>> replace the commas with dots or how do I do it?
>>
>> > I would be very happen if somebody could help me:)
>>
>> Does it have to be done in Matlab? Couldn't you open it in a text
>> editor and use the 'find and replace tool'?
>
> One argument for using matlab would be that the job has
> to be repeated for several files.
>
> These kinds of things often occur because of the locale
> settings on some computer. On one occasion we found this
> very flaw after having delivered hundreds of files to
> a paying client. This was a 24/7 operation, and the flaw
> was discovered just after I had finished my shift. Next
> morning, one guy at the nigh shift had spent 12 hrs
> doing the substitution manually:
>
> - Open the file in an editor
> - Do the find-replace thing
> - Save the file
>
> The only problem was that he had done it in an internal
> database, while it was up to me to hand the corrected
> files to the clients. For bueraucratic reasons, I had
> to either wrap all the newly correcte files (300 if I
> remember correctly) nicley in red tape, or repeat the
> corrections in my archive of files that were already
> 'buearaucratically correct' but which still contained
> the flaw.
>
> So I wrote a matlab script to do the job. It was slow
> compared to the C++ stuff I would have preferred to use,
> but matlab was all I had. I got the job - start writing
> the script to finished conversion - done in less than
> 2 hrs.
>
> Very convenient, when you have a bunch of P-O'd clients
> and bosses hanging around, fuming with rage and
> frustrations.
>
> Rune

Ran into a similar thing once, thankfully our setup had Perl.
perl -p -i.backup -e 's/,/./g' <files>

btw, MATLAB has Perl (!)

  PERL Execute Perl command and return the result.
     PERL(PERLFILE) calls perl script specified by the file PERLFILE
     using appropriate perl executable.

On 1 Jul, 16:07, "Ashish Uthama" <first.l...@mathworks.com> wrote:

> btw, MATLAB has Perl (!)
>
>   PERL Execute Perl command and return the result.
>      PERL(PERLFILE) calls perl script specified by the file PERLFILE
>      using appropriate perl executable.– Skjul sitert tekst –

Thanks for the tip. I didn't know that.

Rune

Rune Allnor wrote:
> On 1 Jul, 16:07, "Ashish Uthama" <first.l...@mathworks.com> wrote:
>
>> btw, MATLAB has Perl (!)
>>
>> PERL Execute Perl command and return the result.
>> PERL(PERLFILE) calls perl script specified by the file PERLFILE
>> using appropriate perl executable.– Skjul sitert tekst –
>
> Thanks for the tip. I didn't know that.

Yeah, the mex setup calls a script -- I discovered that trying to patch
an old version of ML to recognize later version of installed compiler.
I don't know what else may make use of it from TMW.

Unfortunately, I don't know _anything_ about Perl (and am an old-f
retired and not motivated enough to learn enough to try to decipher what
it was doing so didn't get anywhere)...

--

On Wed, 01 Jul 2009 05:41:02 -0400, Jesper Lauridsen
<jesperholst_5@hotmail.com> wrote:

> I have a text file with the following data:
>
> 72008,16193 -0,01 0,02 0,015
> 72008,16375 -0,029 -0,034 -0,034
> 72008,16406 -0,02 -0,054 -0,059
> 72008,16437 -0,024 -0,039 -0,039
> 72008,16467 0,02 -0,044 -0,044
> 72008,16498 -0,005 -0,005 -0,005
> 72008,16529 0,034 0,015 0,01
> 72008,1656 0,01 0,034 0,034
> 72008,16591 0,029 0,01 0,01
>
> I want to load the data in Matlab and replace the comma with dots. Can I
> use the textread function and load it into an cell array and replace the
> commas with dots or how do I do it?

That would be one approach. Did you try it?
(FOPEN, TEXTSCAN, STRREP, STR2DOUBLE)

"Ashish Uthama" <first.last@mathworks.com> wrote in message <op.uwd76lf8a5ziv5@uthamaa.dhcp.mathworks.com>...
> On Wed, 01 Jul 2009 05:41:02 -0400, Jesper Lauridsen
> <jesperholst_5@hotmail.com> wrote:
>
> > I have a text file with the following data:
> >
> > 72008,16193 -0,01 0,02 0,015
> > 72008,16375 -0,029 -0,034 -0,034
> > 72008,16406 -0,02 -0,054 -0,059
> > 72008,16437 -0,024 -0,039 -0,039
> > 72008,16467 0,02 -0,044 -0,044
> > 72008,16498 -0,005 -0,005 -0,005
> > 72008,16529 0,034 0,015 0,01
> > 72008,1656 0,01 0,034 0,034
> > 72008,16591 0,029 0,01 0,01
> >
> > I want to load the data in Matlab and replace the comma with dots. Can I
> > use the textread function and load it into an cell array and replace the
> > commas with dots or how do I do it?
>
> That would be one approach. Did you try it?
> (FOPEN, TEXTSCAN, STRREP, STR2DOUBLE)
>
Hey everybody. Thank you very much for all your comments. The text file I have contain 4 columns and 800000 rows each. I have to replace the commas with dots in all rows and columns. Therefor I would prefer if it is possible til replace the commas, without writing a new file as Rune surgest.

I tried:

>> data = importdata('data_in.dat');
>> dlmwrite('data_out.dat',data,'.')
??? Error using ==> dlmwrite
The input cell array can not be converted to a matrix.

The name of the data file i loaded was data_in.dat


To Samuel:

I have tried to open til file in a text editor and tried to use the search and replace function but the file is too big.

On 1 Jul, 20:53, "Jesper Lauridsen" <jesperhols...@hotmail.com> wrote:

> Hey everybody. Thank you very much for all your comments. The text file I have contain 4 columns and 800000 rows each. I have to replace the commas with dots in all rows and columns. Therefor I would prefer if it is possible til replace the commas, without writing a new file as Rune surgest.

Why is writing a new file a problem? The file can't be *that*
big. 800000 lines of 50 characters each isn't more than 40 MBytes.
I am sure you have disk space for that.

Rune

Rune Allnor <allnor@tele.ntnu.no> wrote in message <b6ceba46-a033-4fb4-a70f-967c776bf8d7@h8g2000yqm.googlegroups.com>...
> On 1 Jul, 20:53, "Jesper Lauridsen" <jesperhols...@hotmail.com> wrote:
>
> > Hey everybody. Thank you very much for all your comments. The text file I have contain 4 columns and 800000 rows each. I have to replace the commas with dots in all rows and columns. Therefor I would prefer if it is possible til replace the commas, without writing a new file as Rune surgest.
>
> Why is writing a new file a problem? The file can't be *that*
> big. 800000 lines of 50 characters each isn't more than 40 MBytes.
> I am sure you have disk space for that.
>
> Rune

You are right Rune. I tried the code you surgest:

f1id = fopen('data_in.dat');
f2id = fopen('repairedfile.dat','w');

while (~feof(f1id))
   s = getl(f1id);
   sidx = find(s==',');
   s(sidx) = '.';
   fprintf(f2id,'%s\n',s);
end

fclose(f1id);
fclose(f2id)'

I got the error:

 Undefined command/function 'getl'.

Error in ==> vibrationer at 5
   s = getl(f1id);

On Wed, 01 Jul 2009 14:53:01 -0400, Jesper Lauridsen
<jesperholst_5@hotmail.com> wrote:

> "Ashish Uthama" <first.last@mathworks.com> wrote in message
> <op.uwd76lf8a5ziv5@uthamaa.dhcp.mathworks.com>...
>> On Wed, 01 Jul 2009 05:41:02 -0400, Jesper Lauridsen
>> <jesperholst_5@hotmail.com> wrote:
>>
>> > I have a text file with the following data:
>> >
>> > 72008,16193 -0,01 0,02 0,015
>> > 72008,16375 -0,029 -0,034 -0,034
>> > 72008,16406 -0,02 -0,054 -0,059
>> > 72008,16437 -0,024 -0,039 -0,039
>> > 72008,16467 0,02 -0,044 -0,044
>> > 72008,16498 -0,005 -0,005 -0,005
>> > 72008,16529 0,034 0,015 0,01
>> > 72008,1656 0,01 0,034 0,034
>> > 72008,16591 0,029 0,01 0,01
>> >
>> > I want to load the data in Matlab and replace the comma with dots.
>> Can I
>> > use the textread function and load it into an cell array and replace
>> the
>> > commas with dots or how do I do it?
>>
>> That would be one approach. Did you try it?
>> (FOPEN, TEXTSCAN, STRREP, STR2DOUBLE)
>>
> Hey everybody. Thank you very much for all your comments. The text file
> I have contain 4 columns and 800000 rows each. I have to replace the
> commas with dots in all rows and columns. Therefor I would prefer if it
> is possible til replace the commas, without writing a new file as Rune
> surgest.
>
> I tried:
>
>>> data = importdata('data_in.dat');
>>> dlmwrite('data_out.dat',data,'.')
> ??? Error using ==> dlmwrite
> The input cell array can not be converted to a matrix.
>
> The name of the data file i loaded was data_in.dat

Did you inspect the data you get from IMPORTDATA. (I would recommend using
FOPEN, TEXTSCAN)

Is it a cell array of strings? If so:

STRREP can be used to replace , with . in strings.
STR2DOUBLE will then convert it to a matrix.

>> strData={ '2,3' '4,5' ; '6,45' '-0,3'}

strData =

     '2,3' '4,5'
     '6,45' '-0,3'

>> strData={ '2,3' '4,5' ; '6,45' '-0,3'};
>> numData=str2double(strrep(strData,',','.'))

numData =

           2.3 4.5
          6.45 -0.3

>> whos
   Name Size Bytes Class Attributes

   numData 2x2 32 double
   strData 2x2 268 cell

On 1 Jul, 21:21, "Jesper Lauridsen" <jesperhols...@hotmail.com> wrote:

> I got the error:
>
>  Undefined command/function 'getl'.
>
> Error in ==> vibrationer at 5
>    s = getl(f1id);

Ah. It should be 'fgetl'.

Rune

Dear Jesper Lauridsen!

> f1id = fopen('data_in.dat');
> f2id = fopen('repairedfile.dat','w');
>
> while (~feof(f1id))
> s = getl(f1id);
> sidx = find(s==',');
> s(sidx) = '.';
> fprintf(f2id,'%s\n',s);
> end
>
> fclose(f1id);
> fclose(f2id);

Just some ideas:
  sidx=find(s==',');
  s(sidx) = '.';
can be written faster as:
  s(findstr(',')) = '.';
This is usually even faster than:
  s(s==',') = '.'

Nevertheless, FGETL and FPRINTF have a remarkable overhead. FGETS and FWRITE can be remarkably faster:
  fid1 = fopen('data_in.dat'); fid2 = fopen('repairedfile.dat','w');
  while 1
    s = fgets(fid1);
    if ischar(s) == 0, break; end % replace FEOF
    fwrite(fid2, strrep(s, ',', '.'), 'uchar');
  end
  fclose(fid1); fclose(fid2);

But if we are on the way, we can drop the WHILE loop and replace the original file immediately:
  fid = fopen('data_in.dat', 'rb+');
  s = fread(fid, inf, 'uchar');
  fseek(fid, 0, -1);
  fwrite(fid, strrep(s, ',', '.'), 'uchar');
  fclose(fid);
The 'b' mode of FOPEN is needed to keep the original line breaks.

Here a surprising problem can appear in FSEEK: For effective multithreading, the operating system can stop FSEEK before it reaches the desired location. This happens more likely on heavy system load and slow (network-) drives, but it is really rare and never reproducible. (NOTE: This is not a Matlab problem.)
Therefore it is safer to check the reply of FSEEK (-1 on failure) or FCLOSE and FOPEN the file again.

Good luck, Jan

On Jul 2, 10:29 am, "Jan Simon" <matlab.THIS_Y...@nMINUSsimon.de>
wrote:
> Dear Jesper Lauridsen!
>
> > f1id = fopen('data_in.dat');
> > f2id = fopen('repairedfile.dat','w');
>
> > while (~feof(f1id))
> >    s = getl(f1id);
> >    sidx = find(s==',');
> >    s(sidx) = '.';
> >    fprintf(f2id,'%s\n',s);
> > end
>
> > fclose(f1id);
> > fclose(f2id);
>
> Just some ideas:
>   sidx=find(s==',');
>   s(sidx) = '.';
> can be written faster as:
>   s(findstr(',')) = '.';
> This is usually even faster than:
>   s(s==',') = '.'
>
> Nevertheless, FGETL and FPRINTF have a remarkable overhead. FGETS and FWRITE can be remarkably faster:
>   fid1 = fopen('data_in.dat'); fid2 = fopen('repairedfile.dat','w');
>   while 1
>     s = fgets(fid1);
>     if ischar(s) == 0, break; end  % replace FEOF
>     fwrite(fid2, strrep(s, ',', '.'), 'uchar');
>   end
>   fclose(fid1); fclose(fid2);
>
> But if we are on the way, we can drop the WHILE loop and replace the original file immediately:
>   fid = fopen('data_in.dat', 'rb+');
>   s = fread(fid, inf, 'uchar');
>   fseek(fid, 0, -1);
>   fwrite(fid, strrep(s, ',', '.'), 'uchar');
>   fclose(fid);
> The 'b' mode of FOPEN is needed to keep the original line breaks.
>
> Here a surprising problem can appear in FSEEK: For effective multithreading, the operating system can stop FSEEK before it reaches the desired location. This happens more likely on heavy system load and slow (network-) drives, but it is really rare and never reproducible. (NOTE: This is not a Matlab problem.)
> Therefore it is safer to check the reply of FSEEK (-1 on failure) or FCLOSE and FOPEN the file again.
>
> Good luck, Jan

Even less complicated:

f1id = fopen('data_in.dat');
f2id = fopen('repairedfile.dat','w');

a=fscanf(f1id,'%c'); % Read everything into string
a=strrep(a,',','.'); % replace commas
fprintf(f2id,'%c',a); % write everything out

fclose(f1id);
fclose(f2id)'

"Jesper Lauridsen" <jesperholst_5@hotmail.com> wrote in message <h2favd$6b6$1@fred.mathworks.com>...
> I have a text file with the following data:
>
> 72008,16193 -0,01 0,02 0,015
> 72008,16375 -0,029 -0,034 -0,034
> 72008,16406 -0,02 -0,054 -0,059
> 72008,16437 -0,024 -0,039 -0,039
> 72008,16467 0,02 -0,044 -0,044
> 72008,16498 -0,005 -0,005 -0,005
> 72008,16529 0,034 0,015 0,01
> 72008,1656 0,01 0,034 0,034
> 72008,16591 0,029 0,01 0,01
>
> I want to load the data in Matlab and replace the comma with dots. Can I use the textread function and load it into an cell array and replace the commas with dots or how do I do it?
>
> I would be very happen if somebody could help me:)

This function is fast, but it overwrites the original file.

function comma2point( strInFileSpec )
    file = memmapfile( strFileSpec, 'writable', true );
    comma = uint8(',');
    point = uint8('.');
    file.Data(( file.Data==comma)' ) = point;
    delete(file)
end

/per

> > I tried:
> >
> >>> data = importdata('data_in.dat');
> >>> dlmwrite('data_out.dat',data,'.')
> > ??? Error using ==> dlmwrite
> > The input cell array can not be converted to a matrix.
> >
> > The name of the data file i loaded was data_in.dat
>
> Did you inspect the data you get from IMPORTDATA. (I would recommend using
> FOPEN, TEXTSCAN)
>
> Is it a cell array of strings?

Right, I see now that your comma isn't a delimiter but a decimal point! Import data assumes that the comma is a delimiter, so if one of your values were missing the decimal point then your row sizes would be different and wouldn't fit into a rectangular matrix; I guess this is why you got your error or, as Rune suggested, because there was something that was interpreted as a string.

> fid = fopen('data_in.dat', 'rb+');
> s = fread(fid, inf, 'uchar');
> fseek(fid, 0, -1);
> fwrite(fid, strrep(s, ',', '.'), 'uchar');
> fclose(fid);

Works for me!

Sorry, missed the last 2 posts. Jesper, maybe you could write a script with 'tic' and 'toc' and let us know which is fastest! Cheers, Sam

On 2 Jul, 10:50, "Samuel Dorrell" <sdorr...@asl-vision.co.uk> wrote:
> Sorry, missed the last 2 posts. Jesper, maybe you could write a script with 'tic' and 'toc' and let us know which is fastest! Cheers, Sam

Interesting question. Below are a number of files that
implement some of the suggestions in this thread.

For moderate file sizes, the results are:

>> testConvertFileNumericFormat(100000,5);
Generate data : 11.03125 seconds
Rune (.m) : 7.92188 seconds
Jan Simon : 1.14063 seconds
TideMan : 7.18750 seconds
Per Isaksson : 0.15625 seconds
Rune (C++) : 0.76563 seconds

For large files, the results are:

>>testConvertFileNumericFormat(1000000,5);
Generate data : 108.60938 seconds
Rune (.m) : 78.62500 seconds
Jan Simon : -- insufficient memory --
TideMan : -- insufficient memory --
Per Isaksson : 1.50000 seconds
Rune (C++) : 8.01563 seconds

The main difference between my C++ algorithm and Per's
algorithm, seems to be that Per loads the whole file
into memory in one read operation, which saves some
time but can hit the ceiling with memory, while I
read the file in many small chunks. This is safer,
what memory is concerned, but takes longer to run.

The C++ file would run a bit faster if the sizes
of the buffers for the input and output streams
were increased.

Rune


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function testConvertFileNumericFormat(nrows,ncols)
    f1name = 'original.txt';
    f2name = 'converted.txt';
    if nargin < 1
    nrows = 1000000;
    end
    if nargin < 2
    ncols = 5;
    end

    t10 = cputime;
    generateTestData(f1name,nrows,ncols);
    t20 = cputime-t10;
    disp(['Generate data :',sprintf('%10.5f',t20),' seconds'])

    t11 = cputime;
    runeConvertFile(f1name,f2name);
    t21 = cputime-t11;
    disp(['Rune (.m) :',sprintf('%10.5f',t21),' seconds'])
    delete(f2name);

    copyfile(f1name,f2name);
    t12 = cputime;
    janSimonConvertFile(f2name);
    t22 = cputime-t12;
    disp(['Jan Simon :',sprintf('%10.5f',t22),' seconds'])
    delete(f2name);

    t13 = cputime;
    tideManConvertFile(f1name,f2name);
    t23 = cputime-t13;
    disp(['TideMan :',sprintf('%10.5f',t23),' seconds'])
    delete(f2name)

    copyfile(f1name,f2name);
    t14 = cputime;
    perConvertFile(f2name);
    t24 = cputime-t14;
    disp(['Per Isaksson :',sprintf('%10.5f',t24),' seconds'])
    delete(f2name);

    t15 = cputime;
    comma2dot(f1name,f2name);
    t25 = cputime-t15;
    disp(['Rune (C++) :',sprintf('%10.5f',t25),' seconds'])
end
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function generateTestData(fn,nrows,ncols)
   fid = fopen(fn,'w');
   data = int32((rand(nrows,2*ncols)-0.5)*2^31);
   for n=1:nrows
       s = [];
       for m=1:ncols
           ss=sprintf('%14d,%-10d',data(n,2*m-1),abs(data(n,2*m)));
           s=[s,ss,' '];
       end
       fprintf(fid,'%s\n',s);
   end
   fclose(fid);
end
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function runeConvertFile(ifn,ofn)
f1id = fopen(ifn);
f2id = fopen(ofn,'w');


while (~feof(f1id))
   s = fgetl(f1id);
   sidx = find(s==',');
   s(sidx) = '.';
   fprintf(f2id,'%s\n',s);
end


fclose(f1id);
fclose(f2id);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function janSimonConvertFile(fname)
  fid = fopen(fname, 'rb+');
  s = fread(fid, inf, 'uchar');
  fseek(fid, 0, -1);
  fwrite(fid, strrep(s, ',', '.'), 'uchar');
  fclose(fid);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function tideManConvertFile(f1name,f2name)
f1id = fopen(f1name);
f2id = fopen(f2name,'w');


a=fscanf(f1id,'%c'); % Read everything into string
a=strrep(a,',','.'); % replace commas
fprintf(f2id,'%c',a); % write everything out


fclose(f1id);
fclose(f2id);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function perConvertFile( strInFileSpec )
    file = memmapfile( strInFileSpec, 'writable', true );
    comma = uint8(',');
    point = uint8('.');
    file.Data(( file.Data==comma)' ) = point;
    delete(file)
end
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
#include <fstream>
#include <algorithm>
#include <iterator>
#include <vector>
#include "mex.h"

using namespace std;

extern void _main();

void mexFunction(
int nlhs,
mxArray *plhs[],
int nrhs,
const mxArray *prhs[]
)
{
  if (nrhs != 2) {
    mexErrMsgTxt("COMMA2DOT requires two input arguments.");
  } else if (nlhs >= 1) {
    mexErrMsgTxt("COMMA2DOT requires no output argument.");
  }

  if (!((mxIsChar(prhs[0]))&&(mxIsChar(prhs[1])))){
    mexErrMsgTxt("Arguments must be strings.");
  }

  if ((mxGetM(prhs[0])!=1)||(mxGetM(prhs[1])!=1)){
    mexErrMsgTxt("Arguments must contain exactly 1 row");
  }

  if ((mxGetN(prhs[0])<1)||(mxGetN(prhs[1])<1)){
    mexErrMsgTxt("Arguments must be non-empty");
  }

  std::wstring f1name((wchar_t*) mxGetData(prhs[0]),
                        mxGetN(prhs[0]));
  std::wstring f2name((wchar_t*) mxGetData(prhs[1]),
                        mxGetN(prhs[1]));

  std::ifstream fin (f1name.c_str(),std::ios_base::binary);
  std::ofstream fout(f2name.c_str(),std::ios_base::binary);
  std::replace_copy(std::istreambuf_iterator<char>(fin ),
                    std::istreambuf_iterator<char>( ),
                    std::ostreambuf_iterator<char>(fout),
                    ',','.');

  return;
}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

On Jul 3, 3:27 am, Rune Allnor <all...@tele.ntnu.no> wrote:
> On 2 Jul, 10:50, "Samuel Dorrell" <sdorr...@asl-vision.co.uk> wrote:
>
> > Sorry, missed the last 2 posts. Jesper, maybe you could write a script with 'tic' and 'toc' and let us know which is fastest! Cheers, Sam
>
> Interesting question. Below are a number of files that
> implement  some of the suggestions in this thread.
>
> For moderate file sizes, the results are:
>
> >> testConvertFileNumericFormat(100000,5);
>
> Generate data :  11.03125 seconds
> Rune (.m)     :   7.92188 seconds
> Jan Simon     :   1.14063 seconds
> TideMan       :   7.18750 seconds
> Per Isaksson  :   0.15625 seconds
> Rune (C++)    :   0.76563 seconds
>
> For large files, the results are:
>
> >>testConvertFileNumericFormat(1000000,5);
>
> Generate data : 108.60938 seconds
> Rune (.m)     :  78.62500 seconds
> Jan Simon     :   -- insufficient memory --
> TideMan       :   -- insufficient memory --
> Per Isaksson  :   1.50000 seconds
> Rune (C++)    :   8.01563 seconds
>
> The main difference between my C++ algorithm and Per's
> algorithm, seems to be that Per loads the whole file
> into memory in one read operation, which saves some
> time but can hit the ceiling with memory, while I
> read the file in many small chunks. This is safer,
> what memory is concerned, but takes longer to run.
>
> The C++ file would run a bit faster if the sizes
> of the buffers for the input and output streams
> were increased.
>
> Rune
>
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> function testConvertFileNumericFormat(nrows,ncols)
>     f1name = 'original.txt';
>     f2name = 'converted.txt';
>     if nargin < 1
>     nrows = 1000000;
>     end
>     if nargin < 2
>     ncols = 5;
>     end
>
>     t10 = cputime;
>     generateTestData(f1name,nrows,ncols);
>     t20 = cputime-t10;
>     disp(['Generate data :',sprintf('%10.5f',t20),' seconds'])
>
>     t11 = cputime;
>     runeConvertFile(f1name,f2name);
>     t21 = cputime-t11;
>     disp(['Rune (.m)     :',sprintf('%10.5f',t21),' seconds'])
>     delete(f2name);
>
>     copyfile(f1name,f2name);
>     t12 = cputime;
>     janSimonConvertFile(f2name);
>     t22 = cputime-t12;
>     disp(['Jan Simon     :',sprintf('%10.5f',t22),' seconds'])
>     delete(f2name);
>
>     t13 = cputime;
>     tideManConvertFile(f1name,f2name);
>     t23 = cputime-t13;
>     disp(['TideMan       :',sprintf('%10.5f',t23),' seconds'])
>     delete(f2name)
>
>     copyfile(f1name,f2name);
>     t14 = cputime;
>     perConvertFile(f2name);
>     t24 = cputime-t14;
>     disp(['Per Isaksson  :',sprintf('%10.5f',t24),' seconds'])
>     delete(f2name);
>
>     t15 = cputime;
>     comma2dot(f1name,f2name);
>     t25 = cputime-t15;
>     disp(['Rune (C++)    :',sprintf('%10.5f',t25),' seconds'])
> end
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
>
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> function generateTestData(fn,nrows,ncols)
>    fid = fopen(fn,'w');
>    data = int32((rand(nrows,2*ncols)-0.5)*2^31);
>    for n=1:nrows
>        s = [];
>        for m=1:ncols
>            ss=sprintf('%14d,%-10d',data(n,2*m-1),abs(data(n,2*m)));
>            s=[s,ss,' '];
>        end
>        fprintf(fid,'%s\n',s);
>    end
>    fclose(fid);
> end
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
>
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> function runeConvertFile(ifn,ofn)
> f1id = fopen(ifn);
> f2id = fopen(ofn,'w');
>
> while (~feof(f1id))
>    s = fgetl(f1id);
>    sidx = find(s==',');
>    s(sidx) = '.';
>    fprintf(f2id,'%s\n',s);
> end
>
> fclose(f1id);
> fclose(f2id);
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
>
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> function janSimonConvertFile(fname)
>   fid = fopen(fname, 'rb+');
>   s = fread(fid, inf, 'uchar');
>   fseek(fid, 0, -1);
>   fwrite(fid, strrep(s, ',', '.'), 'uchar');
>   fclose(fid);
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
>
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> function tideManConvertFile(f1name,f2name)
> f1id = fopen(f1name);
> f2id = fopen(f2name,'w');
>
> a=fscanf(f1id,'%c');  % Read everything into string
> a=strrep(a,',','.');    % replace commas
> fprintf(f2id,'%c',a);   % write everything out
>
> fclose(f1id);
> fclose(f2id);
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
>
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> function perConvertFile( strInFileSpec )
>     file    = memmapfile( strInFileSpec, 'writable', true );
>     comma   = uint8(',');
>     point   = uint8('.');
>     file.Data(( file.Data==comma)' ) = point;
>     delete(file)
> end
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
>
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> #include <fstream>
> #include <algorithm>
> #include <iterator>
> #include <vector>
> #include "mex.h"
>
> using namespace std;
>
> extern void _main();
>
> void mexFunction(
>                  int          nlhs,
>                  mxArray      *plhs[],
>                  int          nrhs,
>                  const mxArray *prhs[]
>                  )
> {
>   if (nrhs != 2) {
>     mexErrMsgTxt("COMMA2DOT requires two input arguments.");
>   } else if (nlhs >= 1) {
>     mexErrMsgTxt("COMMA2DOT requires no output argument.");
>   }
>
>   if (!((mxIsChar(prhs[0]))&&(mxIsChar(prhs[1])))){
>     mexErrMsgTxt("Arguments must be strings.");
>   }
>
>   if ((mxGetM(prhs[0])!=1)||(mxGetM(prhs[1])!=1)){
>     mexErrMsgTxt("Arguments must contain exactly 1 row");
>   }
>
>   if ((mxGetN(prhs[0])<1)||(mxGetN(prhs[1])<1)){
>     mexErrMsgTxt("Arguments must be non-empty");
>   }
>
>   std::wstring f1name((wchar_t*) mxGetData(prhs[0]),
>                         mxGetN(prhs[0]));
>   std::wstring f2name((wchar_t*) mxGetData(prhs[1]),
>                         mxGetN(prhs[1]));
>
>   std::ifstream fin (f1name.c_str(),std::ios_base::binary);
>   std::ofstream fout(f2name.c_str(),std::ios_base::binary);
>   std::replace_copy(std::istreambuf_iterator<char>(fin ),
>                     std::istreambuf_iterator<char>(    ),
>                     std::ostreambuf_iterator<char>(fout),
>                     ',','.');
>
>   return;}
>
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

If you're using Linux, how about:
system(['sed s/,/./g ' infile ' > ' outfile]);

"Jesper Lauridsen" <jesperholst_5@hotmail.com> wrote in message <h2favd$6b6$1@fred.mathworks.com>...
> I have a text file with the following data:
>
> 72008,16193 -0,01 0,02 0,015
> 72008,16375 -0,029 -0,034 -0,034
> 72008,16406 -0,02 -0,054 -0,059
> 72008,16437 -0,024 -0,039 -0,039
> 72008,16467 0,02 -0,044 -0,044
> 72008,16498 -0,005 -0,005 -0,005
> 72008,16529 0,034 0,015 0,01
> 72008,1656 0,01 0,034 0,034
> 72008,16591 0,029 0,01 0,01
>
> I want to load the data in Matlab and replace the comma with dots. Can I use the textread function and load it into an cell array and replace the commas with dots or how do I do it?
>
> I would be very happen if somebody could help me:)

How about
replaceinfile(',', '.', 'myfile.txt')
replaceinfile is in the FEX:
http://www.mathworks.com/matlabcentral/fileexchange/18909

It uses perl (included with MATLAB) and woks pretty fast even with larger files.

On 2 Jul, 22:20, TideMan <mul...@gmail.com> wrote:
> On Jul 3, 3:27 am, Rune Allnor <all...@tele.ntnu.no> wrote:
> > On 2 Jul, 10:50, "Samuel Dorrell" <sdorr...@asl-vision.co.uk> wrote:
>
> > > Sorry, missed the last 2 posts. Jesper, maybe you could write a script with 'tic' and 'toc' and let us know which is fastest! Cheers, Sam
>
> > Interesting question. Below are a number of files that
> > implement  some of the suggestions in this thread.
>
> > For moderate file sizes, the results are:
>
> > >> testConvertFileNumericFormat(100000,5);
>
> > Generate data :  11.03125 seconds
> > Rune (.m)     :   7.92188 seconds
> > Jan Simon     :   1.14063 seconds
> > TideMan       :   7.18750 seconds
> > Per Isaksson  :   0.15625 seconds
> > Rune (C++)    :   0.76563 seconds
...
> If you're using Linux, how about:
> system(['sed s/,/./g ' infile ' > ' outfile]);

I'm using wintel, so I don't have sed.

sed would probably be slower than at least Per's solution,
and maybe also my C++ solution, depending on internal
buffer sizes etc. The reason is that sed has to do the
same read operations, but presumably also uses a more general
string-based search-replace engine, which means it has to
do at least as much work as the character-based routines
for replacing commas with dots.

I would expect sed to work somewhat slower than Jan Simon's
routine. Jan's routine seems to be based on the same
principle as sed, but sed would use more file read operations
to get through the data, thus spending more time.

Rune

Rune Allnor <allnor@tele.ntnu.no> wrote in message <3b23a4a7-7afb-4408-82ad-9d5e0b80319b@24g2000yqm.googlegroups.com>...
> On 2 Jul, 22:20, TideMan <mul...@gmail.com> wrote:
> > On Jul 3, 3:27?am, Rune Allnor <all...@tele.ntnu.no> wrote:
> > > On 2 Jul, 10:50, "Samuel Dorrell" <sdorr...@asl-vision.co.uk> wrote:
> >
> > > > Sorry, missed the last 2 posts. Jesper, maybe you could write a script with 'tic' and 'toc' and let us know which is fastest! Cheers, Sam
> >
> > > Interesting question. Below are a number of files that
> > > implement ?some of the suggestions in this thread.
> >
> > > For moderate file sizes, the results are:
> >
> > > >> testConvertFileNumericFormat(100000,5);
> >
> > > Generate data : ?11.03125 seconds
> > > Rune (.m) ? ? : ? 7.92188 seconds
> > > Jan Simon ? ? : ? 1.14063 seconds
> > > TideMan ? ? ? : ? 7.18750 seconds
> > > Per Isaksson ?: ? 0.15625 seconds
> > > Rune (C++) ? ?: ? 0.76563 seconds
> ...
> > If you're using Linux, how about:
> > system(['sed s/,/./g ' infile ' > ' outfile]);
>
> I'm using wintel, so I don't have sed.
>
> sed would probably be slower than at least Per's solution,
> and maybe also my C++ solution, depending on internal
> buffer sizes etc. The reason is that sed has to do the
> same read operations, but presumably also uses a more general
> string-based search-replace engine, which means it has to
> do at least as much work as the character-based routines
> for replacing commas with dots.
>
> I would expect sed to work somewhat slower than Jan Simon's
> routine. Jan's routine seems to be based on the same
> principle as sed, but sed would use more file read operations
> to get through the data, thus spending more time.
>
> Rune

Hey everybody :) Thank you for your help. Now I got my script to work.
I have another question.

If I have 3 datafiles named y1_filtered.txt, y2_filtered.txt, y3_filtered.txt and I want to load them into matlab automatic how do I do that?

I tried

for i=1:3
        filename=['y',num2str(i),'_filtered.txt'];
        y=load(filename)
end

The loop I made only shows the third file y3_filtered.

If possible I would like to make a vector with all the data from the three files y1_filtered, y2_filtered and y3_filtered.
Each file contains n rows and 1 column:)

On 5 Jul, 13:52, "Jesper Lauridsen" <jesperhols...@hotmail.com> wrote:
> Rune Allnor <all...@tele.ntnu.no> wrote in message <3b23a4a7-7afb-4408-82ad-9d5e0b803...@24g2000yqm.googlegroups.com>...
> > On 2 Jul, 22:20, TideMan <mul...@gmail.com> wrote:
> > > On Jul 3, 3:27?am, Rune Allnor <all...@tele.ntnu.no> wrote:
> > > > On 2 Jul, 10:50, "Samuel Dorrell" <sdorr...@asl-vision.co.uk> wrote:
>
> > > > > Sorry, missed the last 2 posts. Jesper, maybe you could write a script with 'tic' and 'toc' and let us know which is fastest! Cheers, Sam
>
> > > > Interesting question. Below are a number of files that
> > > > implement ?some of the suggestions in this thread.
>
> > > > For moderate file sizes, the results are:
>
> > > > >> testConvertFileNumericFormat(100000,5);
>
> > > > Generate data : ?11.03125 seconds
> > > > Rune (.m) ? ? : ? 7.92188 seconds
> > > > Jan Simon ? ? : ? 1.14063 seconds
> > > > TideMan ? ? ? : ? 7.18750 seconds
> > > > Per Isaksson ?: ? 0.15625 seconds
> > > > Rune (C++) ? ?: ? 0.76563 seconds
> > ...
> > > If you're using Linux, how about:
> > > system(['sed s/,/./g ' infile ' > ' outfile]);
>
> > I'm using wintel, so I don't have sed.
>
> > sed would probably be slower than at least Per's solution,
> > and maybe also my C++ solution, depending on internal
> > buffer sizes etc. The reason is that sed has to do the
> > same read operations, but presumably also uses a more general
> > string-based search-replace engine, which means it has to
> > do at least as much work as the character-based routines
> > for replacing commas with dots.
>
> > I would expect sed to work somewhat slower than Jan Simon's
> > routine. Jan's routine seems to be based on the same
> > principle as sed, but sed would use more file read operations
> > to get through the data, thus spending more time.
>
> > Rune
>
> Hey everybody :) Thank you for your help. Now I got my script to work.
> I have another question.
>
> If I have 3 datafiles named y1_filtered.txt, y2_filtered.txt, y3_filtered.txt and I want to load them into matlab automatic how do I do that?
>
> I tried
>
> for i=1:3
>         filename=['y',num2str(i),'_filtered.txt'];
>         y=load(filename)
> end
>
> The loop I made only shows the third file y3_filtered.

That's because you re-assigne y every time you load
a new file.

If the format of the data is the same in all files,
try something like:

y =[]
for i=1:3
         filename=['y',num2str(i),'_filtered.txt'];
         y=[y;load(filename)];
end

This will collect all the data into one big data set.
If you want to keep the data from each file as a separate
entity, use cell arrays and lead each data set into a
new cell.

Rune