Handling Python program arguments in a json file

I am a Python re-newbie. I would like advice on handling program parameters which are in a f开发者_如何学JAVAile in json format. Currently, I am doing something like what is shown below, however, it seems too wordy, and the idea of typing the same literal string multiple times (sometimes with dashes and sometimes with underscores) seems juvenile - error prone - stinky... :-) (I do have many more parameters!)

#!/usr/bin/env python
import sys
import os
import json                 ## for control file parsing

# control parameters
mpi_nodes = 1
cluster_size = None
initial_cutoff = None
# ...

#process the arguments
if len(sys.argv) != 2:
    raise Exception(
        """Usage:
                run_foo <controls.json>
            Where:
                <control.json> is a dictionary of run parameters
        """
    )

# We expect a .json file with our parameters
controlsFileName = sys.argv[1]
err = ""
err += "" #validateFileArgument(controlsFileName, exists=True)

# read in the control parameters from the .json file
try:
    controls = json.load(open(controlsFileName, "r"))
except:
    err +=  "Could not process the file '" + controlsFileName + "'!\n"

# check each control parameter. The first one is optional
if "mpi-nodes" in controls:
    mpi_nodes = controls["mpi-nodes"]
else:
    mpi_nodes = controls["mpi-nodes"] = 1

if "cluster-size" in controls:
    cluster_size = controls["cluster-size"]
else:
    err += "Missing control definition for \"cluster-size\".\n"

if "initial-cutoff" in controls:
    initial_cutoff = controls["initial-cutoff"]
else:
    err += "Missing control definition for \"initial-cutoff\".\n"

# ...

# Quit if any of these things were not true
if len(err) > 0:
    print err
    exit()

#...

This works, but it seems like there must be a better way. I am stuck with the requirements to use a json file and to use the hyphenated parameter names. Any ideas?

I was looking for something with more static binding. Perhaps this is as good as it gets.

---

Usually, we do things like this.

def get_parameters( some_file_name ):
    source= json.loads( some_file_name )
    return dict(
        mpi_nodes= source.get('mpi-nodes',1),
        cluster_size= source['cluster-size'],
        initial_cutoff = source['initial-cutoff'],
    )

 controlsFileName= sys.argv[1]
 try:
     params = get_parameters( controlsFileName )
 except IOError:
     print "Could not process the file '{0}'!".format( controlsFileName )
     sys.exit( 1 )
 except KeyError, e:
     print "Missing control definition for '{0}'.".format( e.message )
     sys.exit( 2 )

A the end params['mpi_nodes'] has the value of mpi_nodes

If you want a simple variable, you do this. mpi_nodes = params['mpi_nodes']

If you want a namedtuple, change get_parameters like this

def get_parameters( some_file_name ):
    Parameters= namedtuple( 'Parameters', 'mpi_nodes, cluster_size, initial_cutoff' )
    return Parameters( source.get('mpi-nodes',1),
        source['cluster-size'],
        source['initial-cutoff'],
    )

I don't know if you'd find that better or not.

---

the argparse library is nice, it can handle most of the argument parsing and validation for you as well as printing pretty help screens

[1] http://docs.python.org/dev/library/argparse.html

I will knock up a quick demo showing how you'd want to use it this arvo.

---

Assuming you have many more parameters to process, something like this could work:

def underscore(s):
  return s.replace('-','_')

# parameters with default values
for name, default in (("mpi-nodes", 1),):
  globals()[underscore(name)] = controls.get(name, default)

# mandatory parameters
for name in ("cluster-size", "initial-cutoff"):
  try:
    globals()[underscore(name)] = controls[name]
  except KeyError:
    err += "Missing control definition for %r" % name

Instead of manipulating globals, you can also make this more explicit:

def underscore(s):
  return s.replace('-','_')

settings = {}    
# parameters with default values
for name, default in (("mpi-nodes", 1),):
  settings[underscore(name)] = controls.get(name, default)

# mandatory parameters
for name in ("cluster-size", "initial-cutoff"):
  try:
    settings[underscore(name)] = controls[name]
  except KeyError:
    err += "Missing control definition for %r" % name

# print out err if necessary

mpi_nodes = settings['mpi_nodes']
cluster_size = settings['cluster_size']
initial_cutoff = settings['initial_cutoff']

---

I learned something from all of these responses - thanks! I would like to get feedback on my approach which incorporates something from each suggestion. In addition to the conditions imposed by the client, I want something:

1) that is fairly obvious to use and to debug 2) that is easy to maintain and modify

I decided to incorporate str.replace, namedtuple, and globals(), creating a ControlParameters namedtuple in the globals() namespace.

#!/usr/bin/env python

import sys
import os
import collections
import json                 


def get_parameters(parameters_file_name ):
    """
    Access all of the control parameters from the json filename given. A
    variable of type namedtuple named "ControlParameters" is injected
    into the global namespace. Parameter validation is not performed. Both
    the names and the defaults, if any, are defined herein. Parameters not
    found in the json file will get values of None.

    Parameter usage example: ControlParameters.cluster_size
    """
    parameterValues = json.load(open(parameters_file_name, "r"))
    Parameters = collections.namedtuple( 'Parameters',
                    """
                    mpi_nodes
                    cluster_size
                    initial_cutoff
                    truncation_length
                    """
                    )
    parameters = Parameters(
        parameterValues.get(Parameters._fields[0].replace('_', '-'), 1),
        parameterValues.get(Parameters._fields[1].replace('_', '-')),
        parameterValues.get(Parameters._fields[2].replace('_', '-')),
        parameterValues.get(Parameters._fields[3].replace('_', '-'))
    )
    globals()["ControlParameters"] = parameters


#process the program argument(s)
err = ""

if len(sys.argv) != 2:
    raise Exception(
        """Usage:
                foo <control.json>
            Where:
                <control.json> is a dictionary of run parameters
        """
    )

# We expect a .json file with our parameters
parameters_file_name = sys.argv[1]
err += "" #validateFileArgument(parameters_file_name, exists=True)

if err == "":
    get_parameters(parameters_file_name)

    cp_dict = ControlParameters._asdict()
    for name in ControlParameters._fields:
        if cp_dict[name] == None:
            err += "Missing control parameter '%s'\r\n" % name

print err
print "Done"