Import too large csv data file with strings (2024)

Thank you, your code exactly worked for my 850000 lines i needed about 70 seconds, when before i needed 30 minutes. I will check the other commands to see which is the quickest.

Oh, I was out by a factor of 10 on my test data. That explains the difference in size.

From my testing, the longest operation is the strsplit into individual lines. I've just realised that this operation is not actually needed and that the exact same regular expression I wrote can be used on the whole file. You just need to change one option of the regex:

On my machine, it halves the processed time (5 seconds vs 11 seconds for 850000 lines, you need a better computer!).

Also note that the vertcat is also an expensive operation. If you're happy to access your data as a cell array (lines) of cell arrays (tokens), then you can dispense with it.

Finally, note that if a line has 7 or less fields, the regex won't match. That can be worked out by modifying the regex at the expense of more processing time.

I see, yes you are right my time was also reduced, but still i need a better pc. Thank you again

Hello, I am trying to follow this thread and I'm reading through the regex documentation... I don't understand what you are doing with this expression with [^;]* I have a very similar problem, my text is separated by commas and I have seven columns, and I am trying to understand how to use this function similarly.

ADDENDUM OBTW, it might turn out to be faster to use a looping construct and read a smaller subset of the file each pass rather than the whole thing at once...with textscan you can pick up from previous read automagically; textread in this regards always closes the file so it would have to reopen it every time with an updated 'headerlines' argument; probably a losing proposition.

I don't know if this would help or not; you'd just have to 'spearmint to see if less memory requirements per read operation would outperform the alternate.

That are exactly the first 5 lines, i am not interested on the last 3 columns though. As i said there are cases, in which my rows have less than 10 or more than 10 columns, that is why with csvimport function i had my problem solved since those cases were solved through padding or truncation.