Incremental Override for Configuration Data in Hazard Services

Introduction

Base / Site / User and Localization

Configuration data in AWIPS-II is maintained within a Base/Site/User paradigm. This paradigm was first introduced within the Graphical Forecast Editor (GFE), which was originally implemented in the legacy system (the original AWIPS-I). This paradigm proved to be successful, and so was adopted for managing configuration data for the entire breadth of AWIPS-II.

The basic idea behind the Base/Site/User paradigm is as follows: a bare minimum set of default configuration data is delivered with the system software in a set of base/ directories. The default configuration data can be modified through files in site/ and user/ directories. At runtime, when a specific item of configuration data is needed, the software first checks for the configuration data in the appropriate user/ directory. If the desired configuration data is not in the user/ directory, the site/ directory is checked. Finally, if neither the user/ or site/ directory contain the desired configuration data, the default configuration data in the base/ directory is used. We generically refer to any file not in a base/ directory as an override file. Within the context of AWIPS-II, base, site and user are referred to as localization levels.

There are various formats for Hazard Services Configuration files as described in the Focal Point User’s Guide Localization Section.  This document covers the “Incremental Override” formatted files.  Without incremental override, a site/ or user/ configuration file must first be a copy of an entire base/ configuration file, then modified to implement any desired customizations. With incremental override, a site/ or user/ file need only contain those specific configuration items one wants to customize; the rest of the definitions automatically get picked up from the base/ file.

Contents:

The rest of this document will contain five main sections.

• Example Hazard Services Incremental Override Configuration Files: Audience: Focal Points. The specifics of example Hazard Services incremental override configuration files, showing some possible ways to customize those files.
• General Reference Incremental Override Features - Narrative summary of incremental override features:  Audience: Focal Points. Narrative summary of  the features available in the Hazard Services incremental override with several idealized examples.
• Reference guide for keywords that control combination behavior. Audience: Focal Points.
• High Level Code Design: Audience: Developers, Advanced Focal Points. A high level description of the design of the interface to configuration information in Hazard Services.

Example Hazard Services Incremental Override Configuration Files

In the Localization Perspective under the Hazard Services tab (or in directories under ...utility/common_static/base/hazardServices/) you will find the baseline and localization files Hazard Services.  

Under Hazard Categories is one file called HazardCategories.py that is used to organize the existing set of recognized hazards into categories. Here we will show the current contents of that file as it is useful for introducing some of the override concepts:

import collections

HazardCategories = collections.OrderedDict(

      {

      "Convective": [ ("EW","W"), ("SV","W"), ("TO","W"), ("SV","A"), ("TO","A")],

      "Winter Weather": [("BZ", "W"), ("BZ", "A"), ("ZR", "Y"), ("IS", "W"), ("LE", "W"),("LE", "Y"), ("LE", "A"),

                           ("WC", "W"), ("WC", "Y"), ("WC", "A"), ("WS", "W"), ("WS", "A"), ("WW", "Y")],

      "Hydrology": [("FF", "A"), ("FF", "W", "Convective"), ("FF", "W", "NonConvective"),

                      ("FF", "W", "BurnScar"),

                      ("FA", "Y"), ("FA", "A"), ("FA", "W"),

                      ("FL", "A"), ("FL", "W"), ("FL", "Y"), ("HY", "S"), ("HY", "O")],

                       #("FF", "Y")],  # This is not used...        

        "Coastal Flood": [("BH","S"), ("CF", "A"), ("CF", "W"), ("CF", "Y"), ("CF", "S"), ("LS", "A"), ("LS", "W"),

                                  ("LS", "Y"), ("SU", "W"), ("SU", "Y")],

        "Fire Weather": [("FW", "A"), ("FW", "W")],

        "Marine":  [("SE", "A"), ("SE", "W"), ("BW", "Y"), ("GL", "A"), ("GL", "W"), ("HF", "W"), ("HF", "A"),

                    ("LO", "Y"), ("MA", "S"), ("MA", "W"), ("MH", "Y"), ("MH", "W"), ("MF", "Y"), ("MS", "Y"),

                    ("SI", "Y"), ("SC", "Y"), ("SW", "Y"), ("RB", "Y"), ("SR", "A"), ("SR", "W"), ("UP", "A"),

                    ("UP", "Y"), ("UP", "W") ] ,

        "Non Precip": [("AF", "W"), ("AF", "Y"), ("AQ", "Y"), ("AS", "O"), ("AS", "Y"), ("DU", "Y"), ("DS", "W"),

                       ("EH", "W"), ("EH", "A"),("HT", "Y"), ("EC", "W"), ("EC", "A"),  ("FG", "Y"), ("FZ", "W"),

                       ("FZ", "A"), ("HZ", "W"), ("HZ", "A"),("FR", "Y"), ("ZF", "Y"), ("HW", "A"),("HW", "W"),

                       ("LW", "Y"), ("SM", "Y"), ("WI", "Y") ],

        "Tropical": [("TR", "W"), ("TR", "A"), ("HU", "W"), ("HU", "S"), ("HU", "A"), ("HI", "A"), ("HI", "W"),

                     ("TI", "W"), ("TI", "A"), ("TY", "A"),("TY", "W"), ("TS", "A"), ("TS", "W")],          

        }

)

First, note that the name of the variable initialized (HazardCategories) is the same as the name of the file minus the .py extension (HazardCategories.py).  Also note that the top level structure is an OrderedDict; this is fine as this is no more than a standard Python dict that remembers the original order in which elements were added. Finally, we introduce the concept of hazard subtypes here. As with the legacy system WarnGen, there is a separate accounting for flash floods associated with heavy rain ("FF", "W", "Convective") and flash floods not associated with rainfall ("FF", "W", "NonConvective"), which includes warnings for dam failures. Both of these are associated with a VTEC phenomena of FF, and a VTEC significance of W, which means they have a hazard type of FF.W. In Hazard Services, we represent this distinction, where needed, through the hazard subtype. For now, most existing hazard subtypes within Hazard Services have a null subtype designator.

Here is an example of customizing this file based on incremental override. Suppose there was a desire at a WFO to move the extreme cold watch and warning from the Non Precip category to the Winter Weather category. This could be done by creating a Site Level file.  From the Localization Perspective window MB3→ Create Override File → Site.  Include in the following in the new file:

import collections

HazardCategories = collections.OrderedDict(
       {
       "Winter Weather": [("EC", "W"), ("EC", "A")],
       "Non Precip": [ "_override_remove_list_", ("EC", "W"), ("EC", "A")]
       }
)

This illustrates a few things. First, when two lists are encountered at the same namespace, by default they are merged; the list in the override file does not merely displace the list in the base. Next, the default behavior is to add any unique items to the end of the merged list.  Thus, the net effect of this is that the Site Winter Weather list is added to the Base Winter Weather list.

This default behavior can be modified extensively by way of control strings.  In this case the control string used (_override_remove_list_) instructs the list merging logic to remove the items that follow rather than add them.  Thus, the Site Non Precip list will not include the Extreme Cold entries.  More on control strings in the next section.

There may be times when the more conventional non-incremental override behavior is desired even though the file is subject to incremental override.  This is very simple to achieve with a single extra entry placed at the very beginning of the override file content; the form of this entry depends on whether the file is a dictionary or list at the top level.  For a dictionary this extra entry is "_override_replace_" : True,  ; for a list it is "_override_replace_",  .  Then the content that follows can be a complete copy of the base, modified in whatever way is desired.

Under the Hazard Types tab is a file called HazardTypes.py. This file is the primary registry of all the available hazard subtypes. Here we will show parts of that file slightly edited to illustrate some concepts:

OVERRIDE_LOCK =  ['headline', 'combinableSegments', 'includeAll', 'allowAreaChange', 'allowTimeChange', 'expirationTime', True]

HazardTypes = {

         :
        :

       'FF.A' : {'headline': 'FLASH FLOOD WATCH',

              '_override_lock_': OVERRIDE_LOCK,

              'combinableSegments': True,

              'includeAll': True,

              'allowAreaChange': True,

              'allowTimeChange': True,

              'expirationTime': (-30, 30),

              'hazardConflictList': ['FA.A'],

              'warngenHatching': False,

              'ugcType': 'zone',

              'ugcLabel': 'zone',

              'hazardClipArea' : 'cwa',

              'inclusionFractionTest': True,

              'inclusionFraction': 0.1,

              'inclusionAreaTest' : True,

              'inclusionAreaInSqKm' : 1.0,

              'replacedBy': ['FF.W.Convective', 'FF.W.NonConvective'],

              'defaultDuration': 8 * HOURS,

              'durationIncrement': 60,

              'hazardTypeFirstRecommender':'StormTrackTool',

              },

    'FF.W.Convective' : {

              'headline': 'FLASH FLOOD WARNING',

              '_override_lock_': OVERRIDE_LOCK,

              'combinableSegments': False,

              'allowAreaChange': False,

              'allowTimeChange': True,

              'expirationTime': (-10, 10),

              'hazardConflictList': [],

              'warngenHatching': True,

              'ugcType': 'county',

              'ugcLabel': '',

              'hazardClipArea' : 'cwa',

              'hazardPointLimit': 20,

              'durationChoiceList': [ "60 min", "90 min", "120 min", "2 hrs 30 min", "3 hrs", "3 hrs 30 min", "4 hrs", "4 hrs 30 min", "5 hrs", "5 hrs 30 min", "6 hrs", "6 hrs 30 min", "7 hrs", "7 hrs 30 min", "8 hrs" ],

              'defaultDuration': 1 * HOURS,

              'durationIncrement': 15,

              'hazardTypeFirstRecommender':'StormTrackTool',

              'inclusionFractionTest': True,

              'inclusionFraction': 0.1,

              'inclusionAreaTest' : True,

              'inclusionAreaInSqKm' : 1.0,

              'startTimeIsCurrentTime': True,

              'allowTimeExpand': True,

              'allowTimeShrink': False,

              },

    'FL.W' : {'headline': 'RIVER FLOOD WARNING',

              '_override_lock_': OVERRIDE_LOCK,

              'combinableSegments': False,

              'allowAreaChange': False,

              'allowTimeChange': True,

              'expirationTime': (-30, 30),

              'hazardConflictList': [],

              'warngenHatching': False,

              'pointBased': True,

              'ugcType': 'county',

              'ugcLabel': '',

              'hazardClipArea' : 'cwa',

              'inclusionFractionTest': True,

              'inclusionFraction': 0.1,

              'inclusionAreaTest' : True,

              'inclusionAreaInSqKm' : 1.0,

              'replacedBy': ['FL.Y'],

              'defaultDuration': 8 * HOURS,

              'durationIncrement': 60,

              'allowUntilFurtherNotice': True,

              'requirePointId': True

              },

        :
       :

The top level of this data structure is a standard Python dict which contains other dict objects. Note that not every dict has the exact same set of keys; this is perfectly fine as long as the software that uses the configuration data is designed to check for the existence of keys before trying to reference them. Also note that some of the second level dict objects have an '_override_lock_' key. What this does is prevent an override file from changing the contents of any of these objects. This allows for certain configuration data that needs to remain unchanged because it reflects a hard policy directive. Consider the following override files for HazardTypes.py:

(1)                                              
HazardTypes = {                                  
   'FF.A' : { ''defaultDuration': 6 * HOURS, }            

}                    

(2)                                              
HazardTypes = {                                  
  'FF.A' : { 'allowTimeChange': False }

}

(3)                                              
HazardTypes = {                                
 'FF.W.Convective' : "_override_remove_"                    
}

Override file (1) successfully overrides or changes the value of the defaultDuration key for the FF.A hazard.  This is because “defaultDuration” is not in the list of OVERRIDE_LOCK keys in the Base level file.

Override file (2) results in no changes because the definition for the 'FF.A' hazard at the Base level has the “allowTimeChange” key listed in the OVERRIDE_LOCK list.

Override file (3) also cannot change anything, because the FF.W.Convective entry has OVERRIDE_LOCK keys which cannot be removed.  

The Settings directory has one file for each default Setting. Settings can be overridden in a manner similar to HazardTypes.py.  Settings can also be managed through the Settings->Edit menu.

The Product Generator Table directory has one configuration file called ProductGeneratorTable.py. This is where one associates the various product generator modules with the appropriate hazard types. The product generators themselves are class-based files in the Products directory.

If one wanted to create a new generator called My_HYS_ProductGenerator and use it to generate hydrologic statements, one would first need to put the code in a file named My_HYS_ProductGenerator.py in the Product Generators directory.  (See Product Generation sections of the Focal Point User Guide for instructions on creating Product Generators.)

Next, the new My_HYS_ProductGenerator is added via an override to the ProductGeneratorTable.  Here is an example of what that override file would contain:

ProductGeneratorTable = {
       "FLW_FLS_ProductGenerator" : {
           "allowedHazards": [
                "_override_remove_",
                ('HY.S', "Flood5"),
              ]
           },
       "My_HYS_ProductGenerator" : {
           "allowedHazards": [
                ('HY.S', "Flood5"),
              ]
           }
       }

Note that since only one item needed to be deleted from the list of allowedHazards for FLW_FLS_ProductGenerator, the control string _override_remove_ was used. Also note that through the override file we were able to introduce an entirely new key for “My_HYS_ProductGenerator; one is not limited to only changing the contents for existing keys.

General Reference: Incremental Override Features

In this section we will now enumerate the features of the current incremental override capability. Instead of using real world examples, here we will rely on short idealized examples, because the desire is to focus on the functionality rather than the implementation details of any specific set of configuration information.

As described earlier, the incremental override logic in hazard services is capable of dealing with four kinds of files:

• JSON files,
• Python global variable initializations (dictionaries and lists),
• xml, and
• Python class files

With the exception of Python class files, processing any of these data types involves converting the input data for the various localization levels into Python data structures containing JSON serializable objects, invoking the Python class jsonCombine in order to combine the data sets using incremental override principles, and then returning either the raw data structures or the data reformated into the original format. The bulk of this section will discuss this functionality from the aspect of combining JSON files. As previously discussed, the contents of JSON files are most often completely indistinguishable from the right-hand side of a Python variable initialization, so everything said here applies to both. At the end of this section we will say a bit about how this works for xml and Python class files.

Default Override Rules for Dictionaries

Here we show an idealized example that demonstrates the default means by which a base file and an override file are merged to create a combined data structure:

Key “a”: The entry with key "a" demonstrates the default behavior for two dictionaries at the same namespace. If a key only occurs in the base but not the override, or vice versa, that entry is just copied verbatim into the combined result. For entries that have the same key in both the base and override, the override value is used for simple types. For this example, the top level objects also represent dict objects at the same name space. The top level object must be a list in both the base and override or a dict in both the base and override, or else the combine operation will fail.

Key “b”: The entry with key "b" demonstrates the default behavior for two lists at the same namespace. Items in the override that are not already in the base are added to the end of the combined list, but items in the override that are identical to an item already in the base are ignored.

Key “c”: The entries with the key "c" show the behavior when simple types occur at the same name space; the override replaces the base, period.

Keys “d”, “e”: The entries with keys "d" and "e", show what happens when a list in the base occurs at the same namespace as a dict in the override, or vice versa; again the override replaces the base, period.

Keys “a1”, “a2”: The entries at the top level with keys "a1" and "a2" demonstrate that scope and level matter; these are treated as having no relationship at all to the entries with the same key name inside of the dict objects at "a".

Control Strings for Dictionaries

There are several ways to change the default manner in which the two data structures are combined. Anywhere one encounters a string that begins with "_override_", whether that is a key, a value, or a list member, this is treated as something designed to be used to affect the way the two structures are combined. When these strings appear in the base structure, they are generally ignored but stripped; in the override structure they are generally interpreted to affect the combination algorithm and then also stripped. There are some exceptions to this general behavior for override control strings which will be discussed later.

There are only three control strings used within dictionary objects, _override_replace_, _override_remove_ and _override_lock_. Here we show an idealized example that demonstrates how these three override control strings can change the default behavior for combining dictionaries:

First note that, as discussed, the final combined structure has all the control strings stripped out before it is presented to the client.

Key “a”: The entries with key "a" demonstrate how the _override_replace_ control string works. If a key of "_override_replace_" with a value of true occurs in the override structure, that results in the combined structure essentially being a copy of override structure, with all the base entries ignored. As with all instances in these examples, true must be a boolean true, NOT a string "true".  (If the file is Python, then the value is the boolean True.)  Also, as is the case with most of the control strings, the presence of _override_replace_ matters only in the override file; it has no effect in the base.

Key “b”: The entries with key "b" demonstrate how the _override_remove_ control string works. If a key with a value of "_override_remove_" occurs in the override, then that entry is removed from the merged dictionary.

The rest of the examples focus on the control string _override_lock_, and how it interacts with the other control strings. _override_lock_ is exceptional in that it needs to be implemented in the base rather than the override.

Keys “c”, “d”: The entries with keys "c" and "d" demonstrate the most basic functioning of the _override_lock_ control string, if a key of "_override_lock_" with a value of true occurs in a dict object, that has the effect of making that structure permanent; it cannot be changed internally nor removed from the parent.

Key “e”: The entries with key "e" demonstrate an alternate means of implementing the _override_lock_ control string; with a key or list of keys to lock. Note that the keys listed in the base were protected from modification or deletion, but the keys listed in the override were not protected. However, if there were an additional override file to combine (e.g. if there were a Base, a Site, and a User version of this file) then the keys locked in the first override file would apply to the next override file.

Key “f”: The entries with keys "f" show how if specific keys are locked, even if every specific key is locked, that entry CAN be totally removed from the parent.

Default Override Rules for Lists

The concept of namespace is much more ambiguous with lists than with dict objects, and unlike dict objects, lists have no distinction between keys and values. As such, there are many more override control strings needed to effectively manage lists. Furthermore, once you are inside a list whose parent structure is also a list, any accounting of namespace becomes impossible in the current design.

Here we start with an idealized example that demonstrates the default behavior for combining lists:

First note that what appears within a list can be a totally heterogeneous mix of anything that is JSON serializable.

The basic behavior for list combination is as follows: for two lists at the same name space (values for the key "a" here) the combined list at a minimum is a copy of the base list. Then for each list member in the override, it is added to the end of the combined list as long as it is not identical to a member already in the base list.

Here we note a subtle difference between JSON data and Python variable initialization. For JSON data, there is no way to specify whether a dict object is an ordered dict or a standard dict. When one reads in JSON data, one can choose whether to treat all dict objects encountered as ordered or not. For our Hazard Services incremental override logic, we chose the OrderedDict as the default behavior for JSON. This is because it creates fewer instances where the data that comes out is superficially hard to recognize as being the result of the input data because of the quirks of the ordering imposed by the Python dict key hashing algorithm. For Python variable initialization, one can directly choose the ordered or standard dict, depending on the desired behavior.

In the override example here, there are two dictionaries with the same set of key/value pairs, and the JSON combination sees them as different and so adds them both to the combined list. A simple Python initialization would see them as the same, and so add only one to the combined list.

Control Strings for Lists

There are several control strings that do not refer to specific members, but rather control general aspects of combining lists, and these mostly act as binary switches. Here is an idealized example that demonstrates how these control strings work:

Key “a”: The control string _override_prepend_ invokes a behavior where items added to a list from the override are put at the front of the combined list, whereas the control string _override_append_ restores the default behavior of adding them to the end.

Key “b”: The control string _override_additive_ causes all members from an override list at the same name space to be added to the combined list regardless, whereas _override_unique_ restores the default behavior whereby members get added to the combined list only if they are not considered identical to any members of the base list.

Key “c”: The control string _override_by_keys_ causes dictionary list members to be considered identical when they have the same keys, whereas _override_by_content_ restores the default behavior whereby list members must be completely the same to be considered identical.  As demonstrated by the first entry in the lists at key "c", dict objects that are list members and are determined to be identical by an alternate comparison mechanism will be combined.

Key “d”: Finally, the instance of the control string _override_by_key_xxx_ demonstrates the only control string that has a variable part. For this control string, the xxx part varies situationally, and refers to a key for which the value is used to determine whether dict object list members are considered identical. The lists associated with the "d" key in this example demonstrate this. For the first member of the override, the value for the "xxx" key is the same as for a member of the base, and so it is combined with that member. The next member of the override has a totally unique value for its "xxx" key, and so is just appended to the combined list, even though it has the same key set as a member of the base; invoking the _override_by_key_xxx_ behavior is exclusionary with the _override_by_keys_ behavior. After reverting to the default _override_by_content_ behavior, the final member of the override is determined to not be identical to anything in the base and so is also just appended to the combined list, even though it has the same value for its "xxx" key as a base member.

This shows that _override_by_content_ is also exclusionary with the _override_by_keys_ behavior. On the other hand, the behaviors invoked by _override_prepend_/_override_append_, _override_additive_/_override_unique_, and _override_by_key_/_override_by_content_ are relatively orthogonal, and can be combined in various ways.

There are several list control strings that are designed to remove one or more items from the base list at the same namespace. For these entries one or two of the immediately following list members serve as arguments that identify which members to remove. If the argument list members cannot be matched up to items in the base, the operation is ignored and the argument list members are also ignored. Here is an idealized example that shows how these control strings work:

Key “a”: For the lists associated with key "a", the first _override_remove_ successfully removes the entry "a5" from the base. The second _override_remove_, because "a9" is absent from the base, does nothing with "a9". The list member following _override_remove_after_, "a6", is present in the base, and so all list members after that member (but not "a6" itself) are removed. Likewise, _override_remove_before_ results in removing all members before "a3". All these control strings only affect the very next member, so "a0" experiences the default behavior for members in an override list; it is unique so it is added to the end.

Key “b”: For the lists associated with key "b", _override_by_key_bb_ establishes that the value for key "bb" is used to determine when dict object list members are identical. The control string _override_remove_between_ treats the next two items as arguments, so this results in all base list members between the one with an "bb" value of 2 and "b4" being removed. "b0" then gets treated as a new list member to merge in. Similarly, _override_remove_range_ results in everything from "b5" to "b7" being removed.

Key “c”:  For the lists associated with key "c", we show an example of the _override_remove_list_ control string being used. Every override list member after that until the next control string results in an attempt to remove that item from the base. Items that fail to match something in the base will be ignored, but will not terminate the operation. Here we terminate it using the _override_remove_stop_ control string, which is specifically designed for this and to affect nothing else. override_append_ would have worked just as well as this behavior was already in effect and so would have also terminated the list remove operation with no other effect.

There are two control strings that control the overall behavior of a whole list, _override_replace_ and _override_lock_. These must be at the very front of the list to be interpreted, though in the rare case where both are present either can be first or second. As with _override_lock_ in dict objects, this control string needs to be implemented in the base rather than the override.

There is another control string that only applies to the very next item in the list, _override_lock_one_, which as the name implies locks one and only one list member, and is also designed to be implemented in the base. _override_replace_ is implemented in the override like most control strings, and causes the combined list to essentially be a copy of the override list. This idealized example shows how these control strings work.

Key “c”: In the lists associated with the key "c", we show how _override_lock_one_ works, first establishing that the value of the "cc" key should determine whether dict object list members are treated as identical. The second base entry with a "cc" value of 2 is locked, and so the entry in the override that matches up with it cannot affect it. The result of being able to match with something that cannot be changed is not to add that item but to ignore it. At the end we try to remove both "c8" and "c9", but "c8" cannot be removed because it is locked.

There is one thing about locking behavior with lists that needs to be cleaned up. For a dict or list object that resides within a parent dict object, it is very clearly defined that an object that is completely locked internally also cannot be deleted from the parent. For a dict or list object that resides within a parent list object, the behavior is currently inconsistent.

XML files

As was mentioned, the Hazard Services interface to configuration data is interoperable with xml data. The Alerts are configured via an XML file.  One of the challenges involved with xml data is determining when it is best to deal with elements as list members and when it is best to deal with elements as dictionary entries. Here if there are multiple elements with the same name within the same parent element, the default assumption is to treat those elements as list members, otherwise the default assumption is to treat elements as dictionary entries. Here we show one idealized example:

You see the elements named <entry> and <member> get treated as list members by the redundancy rule, and the text within the element <stuff> gets treated as a list based on the attributes supplied. Within the <others> element you see several elements getting treated as dictionary entries because they are completely unique within their namespace. There are also some examples of some non-default behavior being invoked by the use of the override attributes.

Python Classes

NOTE:  Incremental override does NOT apply to Meta Data files, Recommenders, Product Generators or Product Formats which use explicit inheritance of Base classes and override by virtue of being a derived class.

Though not used in baseline Hazard Services, incremental override could interoperate with Python classes.  Here is one quick demo of this. At the following paths we have the indicated Python class file:

at utility/common_static/base/hazardServices/demo/DemoClass.py:

class DemoClass :

   def __init__(self, a, b) :
       self._a = a
       self._b = b

   def method1(self, c) :
       print str(self._a)+str(c)

   def method2(self, d) :
       print str(self._b)+str(d)

and at utility/common_static/site/LLL/hazardServices/demo/DemoClass.py:

class DemoClass :

   def method1(self, c) :
       print str(self._a)+str(self._b)+str(c)

   def method3(self, e) :
       print str(self._a)+str(self._b)+str(e)

Calling the getLocFile() method for the level agnostic path "hazardServices/demo/DemoClass.py" returns the following text:

class DemoClass_Base :

   def __init__(self, a, b) :
       self._a = a
       self._b = b

   def method1(self, c) :
       print str(self._a)+str(c)

   def method2(self, d) :
       print str(self._b)+str(d)

class DemoClass(DemoClass_Base) :

   def method1(self, c) :
       print str(self._a)+str(self._b)+str(c)

   def method3(self, e) :
       print str(self._a)+str(self._b)+str(e)

Instantiating an instance of the DemoClass Python class from this results in the base providing the constructor and method2(), but the override file providing method1() and method3().

Reference guide for keywords that control combination behavior.

As mentioned earlier, any string that begins with "_override_" that is found anywhere within a set of JSON serializable objects subject to incremental override is treated exceptionally. Whether occuring as a list member, a dictionary key, or a dictionary value, it is treated as control string, whose sole purpose is to modify the default behavior of the incremental override functionality. Control strings are never returned to the client as part of the configuration data sets when they are subjected to incremental override.

Here is an alphabetical list of every control string implemented in PV1, along with a description of its functionality.

_override_additive_:

This control string is only interpreted in override files, and is only meaningful as a list member. This control string is exclusionary with the control string _override_unique_, and can apply to any list member that immediately follows up to an occurrence of _override_unique_. All list members that this control string applies to, and are not arguments to another control string, are added to the combined list, regardless of whether they are the same as another member already in the combined list.

_override_append_:

This control string is only interpreted in override files, and is only meaningful as a list member. This control string is exclusionary with the control string _override_prepend_, and can apply to any list member that immediately follows up to an occurrence of _override_prepend_. For any list member that this control string applies to that is appropriate to add to the combined list, this reinvokes the default behavior whereby list members are added to the end of the combined list.

_override_by_content_:

This control string is only interpreted in override files, is only meaningful as a list member, and only applies to list members that are dictionary objects. This control string is exclusionary with the control strings _override_by_key_ and _override_by_keys_, and can apply to any dictionary list member that immediately follows up to an occurrence of _override_by_key_ or _override_by_keys_. The purpose of this control string is to change the way uniqueness is determined among dictionary list members. In this context, a unique list member that is not an argument to another control string will always be added to a combined list. This reinvokes the default behavior whereby dictionary list members must be completely the same to not be treated as unique.

_override_by_key_..._:

This control string is exceptional in that it is the only one that has a variable part. The ... is not literal, but actually refers to an arbitrary key that occurs within dictionary list members.

This control string is only interpreted in override files, is only meaningful as a list member, and only applies to list members that are dictionary objects. This control string is exclusionary with the control strings _override_by_key_ and _override_by_content_, and can apply to any dictionary list member that immediately follows up to an occurrence of _override_by_key_ or _override_by_content_. The purpose of this control string is to change the way uniqueness is determined among dictionary list members. In this context, a unique list member that is not an argument to another control string will always be added to a combined list. For whatever specific key id actually occurs in the ... part of the control string, dictionary list members are considered unique only when they have different values for that key. If a dictionary list member from the override is determined to be non-unique by this test and therefore is not added to the combined list, it undergoes a combination with the base list member that has the same value of that key.

_override_by_keys_:

This control string is only interpreted in override files, is only meaningful as a list member, and only applies to list members that are dictionary objects. This control string is exclusionary with the control strings _override_by_content_ and _override_by_key_, and can apply to any dictionary list member that immediately follows up to an occurrence of _override_by_content_ or _override_by_key_. The purpose of this control string is to change the way uniqueness is determined among dictionary list members. In this context, a unique list member that is not an argument to another control string will always be added to a combined list. This invokes a non-default behavior whereby dictionary list members that have the same set of keys are treated as being unique, regardless of the values associated with those keys.

_override_insert_after_:
This control string is only interpreted in override files, and is meaningful only as a list member. If present, then the list member that immediately follows is considered the "argument" to _override_insert_after_. First, an attempt is made to identify the last member in the combined list that "matches" the "argument". How a match is determined can be impacted by preceding occurrences of _override_by_content_, _override_by_key_, or _override_by_keys_. If a matching member is found, then all new additions to the combined list are inserted after that member. For a series of objects that an occurrence of _override_insert_after_ applies to, only first one is inserted directly after the matching member, and thereafter they are placed right after the last one added.

_override_insert_before_:
This control string is only interpreted in override files, and is meaningful only as a list member. If present, then the list member that immediately follows is considered the "argument" to _override_insert_before_. First, an attempt is made to identify the first member in the combined list that "matches" the "argument". How a match is determined can be impacted by preceding occurrences of _override_by_content_, _override_by_key_, or _override_by_keys_. If a matching member is found, then all new additions to the combined list are inserted immediately before that member.

_override_lock_:

This control string is only interpreted in base files and is meaningful as either a list member or a dictionary key. The purpose of this control string is to identify items in a base object that cannot be changed by things that occur in an override object. If this occurs at the front of a list, or with a value of True in a dictionary, then this prevents the entire object from being changed in any way by an override file. Furthermore, it also prevents the object from being deleted from its parent. "Front" of a list means either first, or second if right after an _override_replace_ control string. For dictionaries, the value can be a key to lock or a list of keys to lock. When locking keys in this manner, the object can still be removed from its parent, unless the list of locked keys contains a boolean True.

_override_lock_one_:

This control string is only interpreted in base files and is meaningful only as a list member. If present, then the one list member that immediately follows cannot be altered or deleted by things that occur in an override object.

_override_multiple_:

This control string is only interpreted in base files and is meaningful as either a dictionary key or a list member.  This control string is used to designate a global merge structure; for a dictionary it is the value of the _override_multiple_ key, for a list it is the next list member.  The global merge structure must be a list or dictionary.  It will be combined with every list member or dictionary value that is of the same type.

_override_null_:
This control string is only interpreted in override files, and is meaningful only as a list member.  This is used when the mere presence of any control string is meant to trigger some behavior, but the desire is to not to invoke any specific control string behavior.  For now, this is used mostly to terminate an _override_remove_list_ operation.

_override_prepend_:

This control string is only interpreted in override files, and is only meaningful as a list member. This control string is exclusionary with the control string _override_append_, and can apply to any list member that immediately follows up to an occurrence of _override_append_. For any list member that this control string applies to that is appropriate to add to the combined list, this invokes a non-default behavior whereby list members are added to the beginning of the combined list. For a series of objects that an occurrence of _override_prepend_ applies to, the first one is prepended in the strictest sense, and thereafter they are placed right after the last one added.

_override_remove_:

This control string is only interpreted in override files, and is meaningful as either a list member or a dictionary value. If present as a value in a dictionary, then that key is removed from the combined dictionary object if it was present in the base. If present as a list member, then the list member that immediately follows is considered the "argument" to _override_remove_. An attempt is made to remove the first list member in the combined object that is the "same" as the "argument". Whether objects are the "same" can be impacted by preceding occurrences of _override_by_content_, _override_by_key_, or _override_by_keys_. If no member in the combined list tests as the "same" as the "argument", then it is completely ignored. Objects can be prevented from being removed by the proper application of _override_lock_ or _override_lock_one_ objects in the base.

_override_remove_after_:

This control string is only interpreted in override files, and is meaningful only as a list member. If present, then the list member that immediately follows is considered the "argument" to _override_remove_after_. First, an attempt is made to identify the last member in the combined list that "matches" the "argument". How a match is determined can be impacted by preceding occurrences of _override_by_content_, _override_by_key_, or _override_by_keys_. If a matching member is found, then every list member in the combined list after the matching member, but not the matching member itself, is removed from the combined list. If a match is not found, then the "argument" is completely ignored. Members can be prevented from being removed by the proper application of _override_lock_ or _override_lock_one_ members in the base.

_override_remove_before_:

This control string is only interpreted in override files, and is meaningful only as a list member. If present, then the list member that immediately follows is considered the "argument" to _override_remove_before_. First, an attempt is made to identify the first member in the combined list that "matches" the "argument". How a match is determined can be impacted by preceding occurences of _override_by_content_, _override_by_key_, or _override_by_keys_. If a matching member is found, then every list member in the combined list before the matching member, but not the matching member itself, is removed from the combined list. If a match is not found, then the "argument" is completely ignored. Members can be prevented from being removed by the proper application of _override_lock_ or _override_lock_one_ members in the base.

_override_remove_between_:

This control string is only interpreted in override files, and is meaningful only as a list member. If present, then the two list members that immediately follow are considered the "arguments" to _override_remove_between_. First, an attempt is made to identify the first member in the combined list that "matches" the first "argument", and the last member in the combined list that "matches" the second "argument". How a match is determined can be impacted by preceding occurrences of _override_by_content_, _override_by_key_, or _override_by_keys_. If two matching members are found, then every list member in the combined list between the two matching members, but not the matching members themselves, are removed from the combined list. If either "argument" fails to match, then both are completely ignored. Members can be prevented from being removed by the proper application of _override_lock_ or _override_lock_one_ members in the base.

_override_remove_list_:

This control string is only interpreted in override files, and is meaningful only as a list member. The presence of _override_remove_list_ means that every list member that follows, up to the next occurrence of a control string, becomes a potential list member to remove from the combined list. For each override list member that _override_remove_list_ applies to, an attempt is made to remove the first list member in the combined object that is the "same". Whether objects are the "same" can be impacted by preceding occurrences of _override_by_content_, _override_by_key_, or _override_by_keys_. If no object is deemed to be the same, then the override list member in question is completely ignored. Lack of a match will not terminate the _override_remove_list_, only  another control string can do that. The control string _override_null_ is available to do this but not affect anything else.

_override_remove_range_:

This control string is only interpreted in override files, and is meaningful only as a list member. If present, then the two list members that immediately follow are considered the "arguments" to _override_remove_between_. First, an attempt is made to identify the first member in the combined list that "matches" the first "argument", and the last member in the combined list that "matches" the second "argument". How a match is determined can be impacted by preceding occurences of _override_by_content_, _override_by_key_, or _override_by_keys_. If two matching members are found, then every list member in the combined list between the two matching members, including the matching members themselves, are removed from the combined list. If either "argument" fails to match, then both are completely ignored. Members can be prevented from being removed by the proper application of _override_lock_ or _override_lock_one_ members in the base.

_override_replace_:

This control string is only interpreted in override files and is meaningful as either a list member or a dictionary key. If this occurs at the front of a list, or with a value of True in a dictionary, then this causes the resultant combined object to be essentially a copy of the override. "Front" of a list means either first, or second if right after an _override_lock_ control string. Individual items in the base, or the whole base object, can be preserved and kept in the combined object by the proper application of _override_lock_ or _override_lock_one_ objects in the base.

_override_unique_:

This control string is only interpreted in override files, and is only meaningful as a list member. This control string is exclusionary with the control string _override_additive_, and can apply to any list member that immediately follows up to an occurrence of _override_additive_. This restores the default behavior whereby members must be considered to be unique to be added to a combined list. How uniqueness is determined can be impacted by preceding occurrences of _override_by_content_, _override_by_key_, or _override_by_keys_

High level Code Design for Configuration File Access.

As Hazard Services was developed, a decision was made to use data encoded in Java Script Object Notation (JSON) as a payload for transferring information between software components within hazard services. Since JSON encoded data is just a string, this makes it possible to successfully transfer information between components without having to know anything about the nature of the information contained. Using JSON as a payload also makes transferring information between different languages easier. This is helpful, because even before Hazard Services came on the scene, the A-II software was already a mix of Java and Python, and Hazard Services has continued with this approach.

Initially, configuration files in Hazard Services were also implemented as JSON data. Eventually it was decided to switch to using Python global variable initializations as the file format for configuration data. This was primarily because the JSON format does not support comments, nor does it support a file include/import mechanism. Except that it starts with a variable name and an equals sign, Python global variable initializations are often virtually indistinguishable from pure JSON. Python uses None where JSON uses null, and Python capitalizes True and False, while JSON leaves them all lower case. Because JSON is still used internally to transfer information between software components, only JSON serializable items are allowed in the global variable initializations that carry Hazard Services configuration data. These items include the simple types of string, integer, float, bool, and None, plus the Python native composite types dict, list and tuple. Once the software that reads Hazard Services configuration data processes it, both lists and tuples end up being presented as lists.

Hazard Services configuration files are never read directly from a disk file, regardless of their format. The contents of these files are requested from EDEX by way of a service. The data for individual files from EDEX are obtained by making calls to a class called LocalFileInstaller. This class allows one to read (or write) the data for one localization file for one specific localization level from EDEX. The LocalFileInstaller class calls other existing A-II classes to do the grunt work of executing the EDEX transaction. LocalFileInstaller provides a stable interface for doing this, allowing the underlying implementation of the transaction to evolve (more on this later).

Please note that A-II has introduced three additional localization levels besides the original base, site, and user from the original GHG. These are configured, workstation, and desk. The configured localization level refers to files that are site specific but are automatically created by the default system installation, rather than being directly customized locally. The workstation localization level refers to files that meant to be used on one particular Cave machine.  The desk localization level refers to files that meant to support a certain class of forecast responsibilities.

The PV2 incremental override recognizes configured and workstation, but not desk.

On top of LocalFileInstaller was created the LocalizationInterface class, which is the primary interface to configuration information for clients. The client provides a path to LocalizationInterface independent of the localization level, and LocalizationInterface will step through each level, request the data for that localization level from EDEX, and combine all the data from the various levels using incremental override principles. LocalizationInterface still has logic that allows it to recognize JSON format files, and can also interoperate with xml files. The client can obtain either JSON serializable Python objects, or can request the combined data be reencoded back into its original format. Furthermore, it can apply incremental override combination to Python class files, but in this case it does not make sense to try to return objects, it will only return the combined Python class.

The jsonCombine class is where all the smarts reside that pertain to combining JSON serializable Python objects using incremental override principles. In fact, the preceding section of this document is largely a reference guide to the various features implemented in the jsonCombine class.

As mentioned, the underlying implementation for conducting EDEX transactions has been evolving. At first, the LocalFileInstaller was almost unchanged from a class of the same name originally posted on the Software Collaboration Portal by Matt Foster. When run outside of the Cave environment, this class as first implemented has always worked very reliably. However, for reasons that are not well understood, this class causes stability problems when invoked repeatedly in a Cave environment. As such, the very lowest level code that directly interacts with EDEX was replaced with Java code, and LocalFileInstaller interoperates with that code using Jep. The logic in the pre-Jep version of LocalFileInstaller was transferred into a class now called AppFileInstaller. If a client calls LocalizationInterface and the Jep based LocalFileInstaller class can be successfully instantiated, then it will use that; otherwise it will switch to using the pure Python AppFileInstaller. This way, clients within Cave are have access to Java based EDEX transactions, whereas clients outside of Cave (where Jep is not available) still have access to configuration information with full incremental override capability.

It is possible that this code which provides access to configuration information will continue to evolve. One possibility is that the version of the logic that runs within Cave will have more and more of its components refactored into Java. That could make it easier to leverage this for an A-II wide capability for providing configuration information with full incremental override capability. Given that Python files will always be one of the file formats that this logic must interoperate with, some parts of this will likely always be coded in Python. Furthermore, there will always be a desire to allow access to configuration information with full incremental override capability for application writers running code outside of Cave. Thus, the logic that supports the pure Python solution is probably never going away entirely, it just may eventually be completely separated out from the version of this logic that runs within Cave.