Memory Error While Calling Genfromtxt Method

Code : import scipy as sp import matplotlib.pyplot as plt data=sp.genfromtxt('data/train.tsv', delimiter ='\t', dtype='string', comments=None, skip_header=1) x = data[:,0] y = da

Solution 1:

You can load it using a np.memmap, which will demand you about 70MB:

import numpy as np
withopen('train.tsv') as f:
    mm = np.memmap('test.memmap', shape=(7395, 27), dtype='|S4000', mode='w+')
    f.next()
    for i, l inenumerate(f):
        mm[i,:] = l.strip().replace('"','').split('\t')

The file is saved when you delete m with del m or when you close the Python console. Maybe you will have to change the mode to r+ after the file is created.

You can work with the memmap array as if it was a normal array, which will allow you to take only the parts of interest.