Pythonic way to extract values from this text file_问答_开发者

I have an output file from a legacy piece of software which is shown below. I want to extract values from it, so that, for example, I can set a variable called direct_solar_irradiance to 648.957, and target ground pressure to 1013.00.

So far, I have been extracting individual lines and processing them like below (repeated many times for the different values I want to extract):

values = lines[97].split()
self.irradiance_direct, self.irradiance_diffuse, self.irradiance_env = values

However, I have now found that extra lines are added to the middle of the output when certain parameters are selected. This means, of course that the 97th line will no longer have the values I need on it.

Is there a good Pythonic way to extract these values, given that there may be extra lines added into the output under certain circumstances? I guess I need to search for known pieces of text in the file, and then extract the numbers referred to by them, but the only ways I can think of doing that are very clunky.

So:

Is there a nice Pythonic way to search for these strings and extract the values that I want?

If not, is there some other way to sensibly do this? (for example, some kind of cool text-file parsing library that I know nothing about).

******************************* 6sV version 1.0B ******************************
*                                                                             *
*                       geometrical conditions identity                       *
*                       -------------------------------                       *
*                       user defined conditions                               *
*                                                                             *
*   month: 14 day :   1                                                       *
*   solar zenith angle:   10.00 deg  solar azimuthal angle:       20.00 deg   *
*   view zenith angle:    30.00 deg  view azimuthal angle:        40.00 deg   *
*   scattering angle:    159.14 deg  azimuthal angle difference:  20.00 deg   *
*                                                                             *
*                       atmospheric model description                         *
*                       -----------------------------                         *
*           atmospheric model identity :                                      *
*               midlatitude summer  (uh2o=2.93g/cm2,uo3=.319cm-atm)           *
*           aerosols type identity :                                          *
*               Maritime aerosol model                                        *
*           optical condition identity :                                      *
*               visibility :  8.49 km  opt. thick. 550 nm :  0.5000           *
*                                                                             *
*                       spectral condition                                    *
*                       ------------------                                    *
*            monochromatic calculation at wl 0.400 micron                     *
*                                                                             *
*                       Surface polarization parameters                       *
*                       ----------------------------------                    *
*                                                                             *
*                                                                             *
* Surface Polarization Q,U,Rop,Chi    0.00000  0.00000  0.00000     0.00      *
*                                                                             *
*                                                                             *
*                       target type                                           *
*                       -----------                                           *
*           homogeneous ground                                                *
*             monochromatic reflectance  1.000                                *
*                                                                             *
*                       target elevation description                          *
*                       ----------------------------                          *
*           ground pressure  [mb] 1013.00                                     *
*           ground altitude  [km] 0.000                                       *
*                                                                             *
*                       plane simulation description                          *
*                       ----------------------------                          *
*           plane  pressure          [mb] 1013.00                             *
*           plane  altitude absolute [km]  0.000                              *
*                atmosphere under plane description:                          *
*                ozone content             0.000                              *
*                h2o   content             0.000                              *
*               aerosol opt. thick. 550nm  0.000                              *
*                                                                             *
*                        atmospheric correction activated                     *
*                        --------------------------------                     *
*                        BRDF coupling correction                             *
*           input apparent reflectance :  0.500                               *
*                                                                             *
*******************************************************************************



*******************************************************************************
*                                                                             *
*                         integrated values of  :                             *
*                         --------------------                                *
*                                                                             *
*       apparent reflectance  1.1287696  appar. rad.(w/m2/sr/mic)  588.646    *
*                   total gaseous transmittance  1.000                        *
*                                                                             *
*******************************************************************************
*                                                                             *
*                         coupling aerosol -wv  :                             *
*                         --------------------                                *
*           wv above aerosol :   1.129     wv mixed with aerosol :   1.129    *
*                       wv under aerosol :   1.129                            *
*******************************************************************************
*                                                                             *
*                         integrated values of  :                             *
*                         --------------------                                *
*                                                                             *
*       app. polarized refl.  0.0000    app. pol. rad. (w/m2/sr/mic)    0.000 *
*             direction of the plane of polarization  0.00                    *
*                   total polarization ratio     0.000                        *
*                                                                             *
*******************************************************************************
*                                                                             *
*                         int. normalized  values  of  :                      *
*                         ---------------------------    开发者_运维技巧                     *
*                      % of irradiance at ground level                        *
*     % of direct  irr.    % of diffuse irr.    % of enviro. irr              *
*               0.351               0.354               0.295                 *
*                       reflectance at satellite level                        *
*     atm. intrin. ref.   background  ref.  pixel  reflectance                *
*               0.000               0.000               1.129                 *
*                                                                             *
*                         int. absolute values of                             *
*                         -----------------------                             *
*                      irr. at ground level (w/m2/mic)                        *
*     direct solar irr.    atm. diffuse irr.    environment  irr              *
*             648.957             655.412             544.918                 *
*                      rad at satel. level (w/m2/sr/mic)                      *
*     atm. intrin. rad.    background  rad.    pixel  radiance                *
*               0.000               0.000             588.646                 *
*                                                                             *
*                                                                             *
*                      sol. spect (in w/m2/mic)                               *
*                                1663.594                                     *
*                                                                             *
*******************************************************************************





*******************************************************************************
*                                                                             *
*                          integrated values of  :                            *
*                          --------------------                               *
*                                                                             *
*                             downward        upward          total           *
*      global gas. trans. :     1.00000        1.00000        1.00000         *
*      water   "     "    :     1.00000        1.00000        1.00000         *
*      ozone   "     "    :     1.00000        1.00000        1.00000         *
*      co2     "     "    :     1.00000        1.00000        1.00000         *
*      oxyg    "     "    :     1.00000        1.00000        1.00000         *
*      no2     "     "    :     1.00000        1.00000        1.00000         *
*      ch4     "     "    :     1.00000        1.00000        1.00000         *
*      co      "     "    :     1.00000        1.00000        1.00000         *
*                                                                             *
*                                                                             *
*      rayl.  sca. trans. :     0.84422        1.00000        0.84422         *
*      aeros. sca.   "    :     0.94572        1.00000        0.94572         *
*      total  sca.   "    :     0.79616        1.00000        0.79616         *
*                                                                             *
*                                                                             *
*                                                                             *
*                             rayleigh       aerosols         total           *
*                                                                             *
*      spherical albedo   :     0.23410        0.12354        0.29466         *
*      optical depth total:     0.36193        0.55006        0.91199         *
*      optical depth plane:     0.00000        0.00000        0.00000         *
*      reflectance I      :     0.00000        0.00000        0.00000         *
*      reflectance Q      :     0.00000        0.00000        0.00000         *
*      reflectance U      :     0.00000        0.00000        0.00000         *
*      polarized reflect. :     0.00000        0.00000        0.00000         *
*      degree of polar.   :         nan           0.00            nan         *
*      dir. plane polar.  :      -45.00         -45.00         -45.00         *
*      phase function I   :     1.38819        0.27621        0.71751         *
*      phase function Q   :    -0.09117       -0.00856       -0.04134         *
*      phase function U   :    -1.34383        0.02142       -0.52039         *
*      primary deg. of pol:    -0.06567       -0.03099       -0.05762         *
*      sing. scat. albedo :     1.00000        0.98774        0.99261         *
*                                                                             *
*                                                                             *
*******************************************************************************
*******************************************************************************



*******************************************************************************
*                        atmospheric correction result                        *
*                        -----------------------------                        *
*       input apparent reflectance            :    0.500                      *
*       measured radiance [w/m2/sr/mic]       :  260.747                      *
*       atmospherically corrected reflectance                                 *
*       Lambertian case :      0.52995                                        *
*       BRDF       case :      0.52995                                        *
*       coefficients xa xb xc                 :  0.00241  0.00000  0.29466    *
*       y=xa*(measured radiance)-xb;  acr=y/(1.+xc*y)                         *

A more complete, perhaps more robust solution will require the use of either a parser using a custom grammer (pyparsing) or some sort of FSM-based processor (TextFSM).

Both options like they'll be non-trivial to use with this output. A (possibly) lighter-weight solution would be to identify each line based on known labels, then extract appropriately (as suggested by other posters).

There are several ways to implement this. I would suggest mapping 'extractor' callables to known line labels, then iterate and call matched extractors. Each callable would take line and a context object/dict as arguments and add attributes to the context as required. Something along the lines of https://gist.github.com/1035938

you could throw your own mini-language, i.e. automate the extraction. I did the following to automate the parsing of a proprietary program-output

# will match in the order written here
tokens = ["num_ref_frames", "Max QP", "Min QP", "Avg QP", "I4x4",
          "I16x16", "SkipZero", "SkipMV", "16x16", "16x8", "8x16",
          "8x8", "8x4", "4x8", "4x4"]

special = ["Quarterpel MVs"]

# this dictionary (hash-table) contains the search string from tokens array
# as well as an array where the first element is the field to extract to
# create matrix array. e.g. 0 = 1st field, 1 = 2nd field, 3 = 3rd field etc.
dict = {tokens[0]:  [1], tokens[1]:  [1], tokens[2]:  [1], tokens[3]:  [1],
        tokens[4]:  [2], tokens[5]:  [2], tokens[6]:  [2], tokens[7]:  [2],
        tokens[8]:  [2], tokens[9]:  [2], tokens[10]: [2], tokens[11]: [2],
        tokens[12]: [2], tokens[13]: [2], tokens[14]: [2],}

Then I simply looped over the input, and for each line checking against the content of token; if match found I did a split according to the dict-entry to extract the correct field.

special above was to handle, well a special variable that required reading from multiple lines.

Update

clone git://gist.github.com/1037403.git to get a copy of the code

usage:
./parser.py all_dec.txt

Hope it helps!

Well if you want a generic parsing library there is pyparsing, but it would probably be overkill in this case.

This appears to be a fairly line-oriented text file, that is not that large in size, so you're best bet would be to loop through each line looking for text that will identify the things you are after.

So something like:

lines = open('file.txt', 'r')
for n, line in enumerate(lines):
    if 'direct solar irr.    atm. diffuse irr.    environment  irr' in line:
        values = lines[n+1].split() # after the next line after this one
        self.irradiance_direct, self.irradiance_diffuse, self.irradiance_env = values

You could then add more if statements and so on as needed to get other data out. Though if you've got a lot of data you'd probably want to generalise the code somewhat. (Probably a dictionary with the text to match as key and a function to call when you match the key).

You may also want to use a regex to match the line, so that you can handle different amounts of white-space better. Otherwise just a one space too many or too few will throw it out.

The best way, IMHO would be to use a mmaped file, and then use regular expression to find what you are looking for.

 text = mmap.mmap(file)
 re.sub(pattern, text)

Mmap module maps a file as it were text, so you can perform pretty much any operations you would perform on a string. And the regex is the best way to search for something. Simple and efficient.

If you need to find specific lines, just handle everything as a string and run specific regular expressions to dig out your gems.

If you need to extract more data, I believe that with a small amount of work you can craft a nice parser for your data. I would use the following functions as a start:

def extract_screens(text):
    """ 
    Returns a list of screens (divided by astericks).
    Each screen is a list of strings stripped from asterisks.
    """
    ...

def process_screen(screen):
    """ 
    Returns a list of screen divisions as tuples: [(heading, body)...]
    heading is a string, body is a list of strings
    blank lines are filtered out.
    """
    ...

By now you should have an indexed list of pieces of text. You can loop through them and execute a simple and specific special parser method for each section.

Tip: Use unit tests to keep yourself sane.