Process large data file in python

Post date: Mar 21, 2015 4:16:12 PM

Process large data file in python

The original question: "I need to process some data that is a few hundred times bigger than RAM. I would like to read in a large chunk, process it, save the result, free the memory and repeat. Is there a way to make this efficient in python?"

credits: Process large data in python

Lazy Method for Reading Big File in Python? -- This is another great posts and comments that I can learn a lot from:

http://stackoverflow.com/questions/519633/lazy-method-for-reading-big-file-in-python?lq=1

Why doesn't Python's mmap work with large files? -- Technical details explanation about mmap.