Reading Tokens From A File In Python 3.x

Is there some way to read in information from a file by token regardless of the formatting? For example, if I am trying to generate a ppm image from an input file and instead of 2

Solution 1:

You could always roll your own file iterator:

class file_tokens:
    def __init__(self, file):
        self.file = file
        self.line = []
    def __iter__(self):
        return self
    def next(self):
        while not len(self.line):
            self.line = self.file.readline()
            if not self.line:
                raise StopIteration
            self.line = self.line.split()
        return self.line.pop(0)

Then use like a normal file:

for token in file_tokens(open(infile)):
    print('Token: ' + token)

Solution 2:

You can use chain.from_iterable, where the iter-able would be line.split() for line in fin:

>>> with open('temp.txt', 'r') as fin:
...      iter = chain.from_iterable(line.split() for line in fin)
...      print(list(iter))
... 
['255', '0', '0', '0', '0', '255']