select_ensembles

select_ensembles.py: select from a list of simulation-ensembles

usage:

  select_ensembles.py [-q] [-0] [-S] [-n N] [-k key] [-o key] MODE CSVFILE

where: MODE is "top", "bottom", or "mix" CSVFILE is a CSV file containing yields and experiment names (it is as generated by reduce-drms.py) and:

  -n N tells how many rows to select.  Default is 10.  Non-negative, or T for all rows.
  -k key gives the "value key" to use for "top", "bottom", or "mix".  Default is first key.
  -o key gives the "output key" which is printed to stdout
  -S means to use Sandbox filename conventions to get CSVFILE
  -0 means to use \0 as the item separator for output keys
  -t THRESH means to output only records where the value key is <= THRESH
  -T THRESH means to output only records where the value key is > THRESH
  -q means to exit quietly if CSVFILE is not present

The interpretation of MODE is: top (bottom): select the rows containing the top (resp. bottom) N values of KEY, with values compared numerically if possible, and alphabetically otherwise. mix: select an arbitrary mix of N rows, with the rows selected according to the md5 hash of the value of KEY. This provides stability across invocations if rows are added to the CSVFILE between runs, provided the KEY is unchanged.

For simplicity, giving [-S] flags that the csv file will be derived using the directory named in CSVFILE, and sandbox conventions ("reduce-info.csv" for an Ensemble, and "reduce-yield-plus.csv" for .exp/.fam).

If you want to select all records with value key no more than 2, use "-t 2 -n T", so that there is no upper limit on the number of keys selected.

In addition, the pseudokey _index (for -k or -o) gives the ordinal number of the CSV row. Also, for MODE of mix, when -k KEY is used, the pseudokey _KEY is available as output, and yields the md5 hash of the value of KEY for that row (e.g., for -k experiment, the pseudokey is: _experiment).

Typical usage: util/select_ensembles.py -o experiment -k chars_earth_unique top sims/HabEx_4m_TSDDold_DD_TF17_promo6_20190123.exp/reduce-yield-plus.csv