Process large data file in python
Post date: Mar 21, 2015 4:16:12 PM
Process large data file in python
The original question: "I need to process some data that is a few hundred times bigger than RAM. I would like to read in a large chunk, process it, save the result, free the memory and repeat. Is there a way to make this efficient in python?"
credits: Process large data in python
Lazy Method for Reading Big File in Python? -- This is another great posts and comments that I can learn a lot from:
http://stackoverflow.com/questions/519633/lazy-method-for-reading-big-file-in-python?lq=1
Why doesn't Python's mmap work with large files? -- Technical details explanation about mmap.