Splitting 25mb .txt file into smaller files using text delimiter

Regards, SO

I am new to python and Perl. I have been trying to solve a simple problem and getting tied in knots with syntax. I hope someone has the time and patience to help. I have a 25mb file in ".txt" format which contains news-wire articles going back to 1970. Each news story is concatenated to the next, with only the "Copyright" statement to delimit. Each news story starts with "Item XX of XXX DOCUMENTS". There are certain metadata that are repeated throughout, I will use these for tagging later on.

I wish to split this 25mb file into separate .txt files, each containing one news story (i.e. the text between "DOCUMENTS" and "Copyright", saving each with a different name (obviously).

I am trying to 1 ) open the file... 2) iterate over lines in the file checking for the eof delimiter, and if it is not present writing the line to a list 3)write that list to a seperate small file.

I'm having big problems with changing filenames using the counter, and how do I make Python start from where I left off, is the "seek" function appropriate?

so far I have been trying this approach, completely unsuccessfully:

myfile = open ("myfile.txt", 'r')
filenumber = 0
for line in myfile.readline():  
    filenumber += 1    
    w=0  
    while myfile.readline() != '\s+DOCUMENTS\s*\n'  
    ### read my line into a list  
    mysmallfile()['w'] 开发者_StackOverflow= [myfile.readline()]  
    w += 1  
    output = open('C:\\Users\\dunner7\\Documents\###how do I change the filename      each     iteration???', 'w')  
    output.writelines(mysmallfile)   
    ###go back to start.   

Thank you for your time and patience.

RD

Here is a sample of the text file:

                           1 of 575 DOCUMENTS

                         The Washington Daybook

                            January 28, 2011

Health and Human Services Department (HHS); Food and Drug Administration (FDA) (F.R. Page 72832) holds a meeting of the Neurological Devices Panel of the Medical Devices Advisory Committee to discuss and make recommendations regarding the possible reclassification of devices indicated for use in electroconvulsive therapy, January 27-28.

TIME: 8 a.m.

LOCATION: Hilton Washington DC North/Gaithersburg, Ballroom, 620 Perry Parkway, Gaithersburg, Md.

CONTACT: James Engles, 800-741-8138 [Note: Use the code: 3014512513, when calling for information.]

LOAD-DATE: November 28, 2010

LANGUAGE: ENGLISH

TYPE: Meeting

       Copyright 2011 Federal Information and News Dispatch, Inc.


                           2 of 575 DOCUMENTS

                         The Washington Daybook

                            January 27, 2011

Health and Human Services Department (HHS); Food and Drug Administration (FDA) (F.R. Page 72832) holds a meeting of the Neurological Devices Panel of the Medical Devices Advisory Committee to discuss and make recommendations regarding the possible reclassification of devices indicated for use in electroconvulsive therapy, January 27-28.

TIME: 8 a.m.

LOCATION: Hilton Washington DC North/Gaithersburg, Ballroom, 620 Perry Parkway, Gaithersburg, Md.

CONTACT: James Engles, 800-741-8138 [Note: Use the code: 3014512513, when calling for information.]

LOAD-DATE: November 28, 2010

LANGUAGE: ENGLISH

TYPE: Meeting

       Copyright 2011 Federal Information and News Dispatch, Inc.


                           3 of 575 DOCUMENTS


                              FNS DAYBOOK

                       January 12, 2011 Wednesday
                              FUTURE EVENTS

EVENT: MEETING - HEALTH AND HUMAN SERVICES DEPARTMENT (HHS); FOOD AND DRUG ADMINISTRATION (FDA) (F.R. PAGE 72832); LOCATION: Hilton Washington DC North/Gaithersburg, Ballroom, 620 Perry Parkway, Gaithersburg, Md. -- January 27, 2011 8:00 am

SECTION: FEDERAL AGENCIES AND DEPARTMENTS - FUTURES

LENGTH: 72 words

SUBJECT: Health and Human Services Department (HHS); Food and Drug Administration (FDA) (F.R. Page 72832) holds a meeting of the Neurological Devices Panel of the Medical Devices Advisory Committee to discuss and make recommendations regarding the possible reclassification of devices indicated for use in electroconvulsive therapy, January 27-28.

CONTACT: James Engles, 800-741-8138 [Note: Use the code: 3014512513, when calling for information.]

LOAD-DATE: January 10, 2011

LANGUAGE: ENGLISH

PUBLICATION-TYPE: Event Schedules

                  Copyright 2011 Federal News Service
                          All Rights Reserved

---

Something like that:

filenumber = 0
outfile = None
with open('source_file.txt') as f:
    for line in f:
        if line.strip() == 'DOCUMENTS':
            filenumber += 1
            outfile = open('result%03d.txt' % filenumber, 'w')
        elif line.strip().startswith('Copyright') and outfile:
            outfile.close()
            outfile = None
        elif outfile:
            outfile.write(line)
if outfile:
    outfile.close()

I had to guess lots of things because I don't know exacly how the file looks like. Post the file if you have further problems.

---

Several problems along the way:

• You're using myfile.readline() in both the loop iterator and when writing to a small file -- just use line when you want to know the contents of the current line
• The myfile.readline() != 'string' bit is always going to fail, this isn't actually a regexp match.
• mysmallfile()['w'] looks overly clever (if you're using the mysmallfile() function to return an array or dictionary indexed with 'w' to retrieve an L-value(!)) or like you're just way off course here :)
• You open a file, write a line into it, but then never close the file. This is recipe for failing after 1000 lines when you run out of open file descriptors.

There's a lot of cleanup that went into nosklo's suggestion, I hope my post can help you understand how he got to his. :)