3.3 Product Generation Examples contents ⇑ ↑ ^ v ↓ ⇓
IMPORTANT!!!
Anyone that hopes to make modifications to the generated product text must be able to identify which product part each piece of text in the product is attached to. The most straightforward way is to make a temporary user override version of NWS_Base_Formatter.py containing the code that follows. Changes from the base version are in green. This will also route the product part debug information into the flat file "/tmp/printDebugProductParts.txt" (sample here); this allows a user without access to a full blown ADE to still be able to do this kind of customization. Even with an ADE, sometimes it is inconvenient trying to separate the product part debug information from the ‘firehose’ of other diagnostics that can land in the eclipse console.
class Format(FormatTemplate.Formatter): |
If for some reason it is not desired to route these diagnostics to a temporary file, one can override the method _productPartsTmpFile() to return None. If there is a desire to route these diagnostics to a different file, one can override _productPartsTmpFile() to return a different file path.
As has been mentioned, most product parts follow a paradigm where there is one instance of the part method in the appropriate formatter (or the NWS_Base_Formatter.py super class) and another instance in one of the LevelMethods modules. Generally, the instance in a LevelMethods module generates the text content of the part. The instance in the formatter directly interfaces with the product parts executive logic, and usually calls a method called processPartValue(). It is the job of processPartValue() to determine if useable text already exists for the part; the existing text is used if available, and otherwise the content method is called to get the text for the part.
Two product parts that notably diverge from this paradigm are firstBullet and attribution. The structure of the code that handles these parts is much more complicated. This is discussed in detail in section 4.4.
3.3.0 Product Editor configuration contents ⇑ ↑ ^ ─ + v ↓ ⇓
In this we discuss how to configure the characteristics of individual product parts. To aid in this discussion, we will show how one might customize Hazard Services such that a product part that previously was not available to edit in the Product Editor is configured to be editable. Consider a Flash Flood Warning issued for an emergency situation; part of such a warning might look like the following:
BULLETIN - EAS ACTIVATION REQUESTED
Flash Flood Warning
National Weather Service Omaha/Valley NE
437 PM CDT Thu Sep 7 2017
...FLASH FLOOD EMERGENCY FOR AREAS NEAR FREMONT…
The National Weather Service in Omaha/Valley has issued a
* Flash Flood Warning for…
Currently in hazard services, the text in blue is not available for editing; here we show how to customize Hazard Services such that it is available for editing. First one needs to identify which product part this piece of the product is in as described in section 3.3. The product part so identified is emergencyHeadline. In order to make this product part editable, one needs to override its entry in ProductEditorFields.py. This file is subject to incremental override and so its site override would only need to contain what we want to change:
ProductEditorFields = { |
This is all that is required in order to make this product part editable in the Product Editor. One downside of this is that if the **SELECT FOR FLASH FLOOD EMERGENCY** checkbox is not activated in the Hazard Information Dialog, then this editable field will still always appear, albeit empty. Therefore, if one is approaching this by only overriding ProductEditorFields.py, one must resist any urge to set "required" to True. Otherwise it will then be impossible to actually issue a product without some content for this headline, which will be problematic for issuing non-emergency flash flood hazards.
One can learn more about what these various entries mean by inspecting the base level version of ProductEditorFields.py.
By overriding another file besides ProductEditorFields.py, it is possible to have the set of product parts used adapt to whether the Flash Flood in question is for an emergency situation. This requires that one also override IBW_FFW_FFS_Formatter.py(changes from base in green):
IBW_FFW_FFS_Formatter.py:
class Format(NWS_Base_Formatter.Format): |
What is happening here is that we first call the existing product part code and record the result instead of directly returning it. Then we search recursively through that returned structure for the specific list element containing the part of interest, which is 'emergencyHeadline'. If that structure is found, then we determine if the hazard is for a flash flood emergency. Finally, if not for an emergency, we extract the 'emergencyHeadline' product part before returning the product parts list.
Further note that with this override in place, it then becomes practical to set the "required" state for the "emergencyHeadline" product part to True if desired.
3.3.0.1 “Aliasing” an existing Product Part module contents ⇑ ↑ ^ v ↓ ⇓
Consider a situation where we want to modify how we present a product part in the Product Editor for one situation, but we do not need to modify the wording of the text that is generated. To demonstrate this concept, we will continue to leverage the example from the previous section (3.3.0). Here we assume that when an FF.W hazard is followed up (e.g. we issue an update with a VTEC Action of CON) we do not want the emergency headline to be editable, even though for the original issuance it will continue to be editable. This has the effect of rendering the emergency headline text “locked” for follow ups.
It is impossible for the same product part to have different characteristics in the Product Editor in different situations; this must be addressed by adding a new product part. However, it is possible to “alias” a product part with an extremely small override. In this example we “alias” the emergencyHeadline product part to the product part of emergencyHeadlineFFS with this minor override to NWS_Base_Formatter.py:
class Format(FormatTemplate.Formatter): |
We could have just as well added this one method to the existing override of IBW_FFW_FFS_Formatter.py. This demonstrates that there is a good deal of flexibility in how one implements such an “alias”.
Then, in order to have this product part be used for updating an FF.W hazard (e.g. issuing an FFS), we make the following override to HydroProductParts.py:
class HydroProductParts(object): |
Given that the desire is to have the new emergencyHeadlineFFS product part be non-editable, which is the default, there is no need for any further override of ProductEditorFields.py.
3.3.1 Creating a completely new Product Part contents ⇑ ↑ ^ ─ + v ↓ ⇓
This section will walk through the process of adding a brand new product part to an existing product. To facilitate this, we will use adding a part called “emergencyContacts” to FFW products (flash flood warnings) as an example. (Note that this kind of thing could just as easily be added as an additional call to action instead.)f
Note that for this example we start by adding this product part in an override to ProductEditorFields.py. If the default behavior of not presenting this new product part in the product editor was acceptable, then this step could be omitted.
However, for this example we assume there was a desire to make the new product part editable, and so the site override for ProductEditorFields.py might look like the following:
ProductEditorFields = { |
3.3.1.1 Update HydroProductParts contents ⇑ ↑ ^ ─ + v ↓ ⇓
The next step is to create a site level override for HydroProductParts.py where we add the newly defined product part to the list of parts to include for one or more specific text products. For this example (adding “emergencyContacts” to FFW products) the site level override would look like the following (changes from base in green):
class HydroProductParts(object): |
3.3.1.2 Update Formatter contents ⇑ ↑ ^ ─ + v ↓ ⇓
The next step is to override the corresponding formatter with a method that generates text for the new product part. Note that with the current code organization, this is typically done with a method in a formatter, which is the interface to the product parts executive logic, and a method in one of the ‘LevelMethods’ modules, which actually generates the content.
For this example we will override IBW_FFW_FFS_Formatter.py (although this could also be done with NWS_Base_Formatter.py) and SectionLevelMethods.py. The site overrides of these files supporting this change can look like the following:
IBW_FFW_FFS_Formatter.py:
class Format(NWS_Base_Formatter.Format): def emergencyContacts(self, sectionDict): contacts = self.processPartValue(sectionDict, 'emergencyContacts', self.sectionMethods.emergencyContacts, sectionDict) startText = '' # Uncomment these lines if you want bullet format # act = self.getAction(sectionDict.get('vtecRecord',{})) # if act in ['NEW', 'EXT']: # startText = '* ' if self._testMode: startText += self._stdTestPhrase return self.getFormattedText(contacts, startText=startText, endText='\n\n') |
SectionLevelMethods.py:
class SectionLevelMethods(object): def emergencyContacts(self, sectionDict): partText = 'In case of emergency contact your local law enforcement office.' return partText |
An example of the modified product is shown below. The new product part is colored blue.
WGUS53 KOAX 122123
FFWOAX
NEC023-037-053-155-130030-
/O.NEW.KOAX.FF.W.0022.180712T2123Z-180713T0030Z/
/00000.0.ER.000000T0000Z.000000T0000Z.000000T0000Z.OO/
BULLETIN - EAS ACTIVATION REQUESTED
Flash Flood Warning
National Weather Service Omaha/Valley NE
423 PM CDT Thu Jul 12 2018
The National Weather Service in Omaha/Valley has issued a
* Flash Flood Warning for...
Northeastern Butler County in east central Nebraska...
Northwestern Dodge County in east central Nebraska...
Northwestern Saunders County in east central Nebraska...
Eastern Colfax County in northeastern Nebraska...
* Until 730 PM CDT.
* At 423 PM CDT, Doppler radar indicated thunderstorms producing
heavy rain across the warned area. Flash flooding is ongoing or
expected to begin shortly.
HAZARD...Flash flooding caused by heavy rain.
SOURCE...Doppler radar.
IMPACT...Flooding of small creeks and streams, urban areas,
highways, streets and underpasses as well as other
drainage and low lying areas.
* Some locations that will experience flash flooding include...
Schuyler, Scribner, Rogers, 7 Miles North Of North Bend and 9
Miles North Of Schuyler.
In case of emergency contact your local law enforcement office.
PRECAUTIONARY/PREPAREDNESS ACTIONS...
Turn around, don't drown when encountering flooded roads. Most flood
deaths occur in vehicles.
&&
LAT...LON 4167 9706 4170 9659 4154 9660 4137 9708
FLASH FLOOD...RADAR INDICATED
$$
3.3.2 Removing a section from a product. contents ⇑ ↑ ^ ─ + v ↓ ⇓
Here we will give an example of how one can remove a section from an existing text product. For this example we will remove the Crest Comparison/Flood History section from a Flood Warning (FL.W) product. There are two subtasks here. First, removing the piece of FL.W metadata that posts the items in the Hazard Information Dialog related to this section. Second, modifying the text generation such that this section is not included in the FL.W text product.
This starts by noting that the tab in the Details portion of the Hazard Information Dialog related to this section is labeled ‘Crest Comparison’, and that the text entry field in the Product Editor related to this section is labeled ‘Flood History Bullet’. So to find the files one needs to make site overrides for, run these commands (see Introduction for more information about searching code for functionality):
> cd /awips2/edex/data/utility/common_static/base/HazardServices
> find . -name '*.py' -exec grep 'Crest Comparison' '{}' \; -print
> find . -name '*.py' -exec grep 'Flood History' '{}' \; -print
This actually identifies five files; MetaData_FL_W.py, CommonMetaData.py, SectionLevelMethods.py, HydroProductParts.py, and ProductEditorFields.py.
One can inspect MetaData_FL_W.py and CommonMetaData.py to learn that CommonMetaData.py formats what the Crest Comparison tab looks like, whereas MetaData_FL_W.py decides whether to include this tab in the metadata. (This is a very common paradigm for the various sections of the Hazard Information Dialog). Therefore, MetaData_FL_W.py is what we override to remove the Crest Comparison tab.
One needs to recall that formatters control how text looks, whereas product parts control which sections are included in the text. ProductEditorFields.py controls how product parts are presented in the Product Editor and has nothing to do with whether they are included in any given product. In SectionLevelMethods.py the content of the part is determined, but not whether it appears in any given product. Therefore the file we need to override to remove the Flood History section from the text product is HydroProductParts.py.
What follows are the contents of the site overrides needed to support removing this section, with the code changed from the base version of the method in question in green.
HydroProductParts.py:
class HydroProductParts(object): def sectionPartsList_FFA_FLW_FLS_point(self, vtecRecord): action = self.tpc.getVtecAction(vtecRecord) phensig = vtecRecord['phensig'] partsList = [ 'setUp_section', 'attribution_point', 'firstBullet_point', ] if action not in ['ROU', 'CAN', 'EXP']: partsList.append('timeBullet') if action in ['CAN', 'EXP']: partsList += [ 'observedStageBullet', 'recentActivityBullet', 'forecastStageBullet', ] else: # Oct 2017: Add action stage for FL.Y. # Uncomment lines to get action and/or bankfull stage for # situations that don't include those by default. if phensig in ['FL.Y']: partsList += [ 'observedStageBullet', # 'bankFullStageBullet', 'actionStageBullet', 'floodStageBullet', 'recentActivityBullet', 'forecastStageBullet', 'pointImpactsBullet', ] else: partsList += [ 'observedStageBullet', # 'bankFullStageBullet', # 'actionStageBullet', 'floodStageBullet', 'floodCategoryBullet', 'recentActivityBullet', 'forecastStageBullet', 'pointImpactsBullet', ] # According to the Directives (NWSI 10-922, Section 4), # the "Flood History" bullet is not allowed for FL.A. # Rather than check for FL.A, however, Mark Armstrong # in an email on 9/29/2016 has requested the below # logic # Remove from FL.W too for this override # if phensig in ['FL.W'] and action not in ['CAN', 'EXP']: # partsList.append('floodHistoryBullet') return partsList |
MetaData_FL_W.py:
class MetaData(CommonMetaData.MetaData): |
3.3.3 Disabling ability to issue a hazard type. contents ⇑ ↑ ^ ─ + v ↓ ⇓
At times it may be desirable to disable the ability to issue a hazard type without removing all knowledge of that hazard type from Hazard Services. This allows the forecaster to still view hazards of that type from other WFOs for situational awareness purposes. For this example we disable the ability to issue the hazard types river flood watch (FL.A), river flood advisory (FL.Y), and areal flood advisory (FA.Y). This will involve three steps. First we remove the capability for the River Flood Recommender to recommend products of the type FL.A and FL.Y. This is done with a site override of RiverFloodRecommender.py. The override being provided here overrides three methods; defineDialog(), filterHazardsByType(), and setHazardType(). This override assumes that, for river gauges where the default code would normally recommend an FL.A or FL.Y, there is still a desire to be able to get an HY.S recommendation. If instead there were a desire to completely throw away any FL.A or FL.Y recommendations, then the override for the method setHazardType() could be left out. This is a fairly large override, but only because we are overriding multiple substantial methods. The changes themselves, in the customary green, are very minor.
RiverFloodRecommender.py:
class Recommender(RecommenderTemplate.Recommender): def defineDialog(self, eventSet, **kwargs): """ @return: A dialog definition to solicit user input before running tool """ methodInput = None jMethodInput = kwargs.get("methodInput", None) chosenPointID = None if jMethodInput is not None: methodInput = JUtil.javaMapToPyDict(jMethodInput) chosenPointID = methodInput.get(CHOSEN_POINT_ID, None) self.siteId = methodInput.get('siteID') dialogDict = {"title": "Flood Recommender"} runType = {} runType["fieldType"] = "RadioButtons" runType["fieldName"] = RUN_TYPE_KEY runType["label"] = "" runType["choices"] = [{'identifier': RUN_TYPE_VAL_FULL, 'displayString': 'Create/update/end hazards for chosen river points'}, {'identifier': RUN_TYPE_VAL_REFRESH, 'displayString': 'Refresh existing hazards for chosen river points'}] runTypeGroup = { "fieldName": runType.get('fieldName') + 'Group', "fieldType":"Group", "label": "Recommender Run Type", "expandHorizontally": False, "expandVertically": False, "fields": [runType] } # Only add the filter widgets when running for all points if chosenPointID is None: choiceFieldDict = {} choiceFieldDict["fieldType"] = "RadioButtons" choiceFieldDict["fieldName"] = "forecastType" choiceFieldDict["label"] = "" #choiceFieldDict["choices"] = ["Watch", "Warning", "Advisory", "ALL"] choiceFieldDict["choices"] = ["Warning"] choiceFieldGroup = { "fieldName": choiceFieldDict.get('fieldName') + 'Group', "fieldType":"Group", "label": "Type", "expandHorizontally": False, "expandVertically": False, "fields": [choiceFieldDict] } includeNonFloodPointDict = {} includeNonFloodPointDict["fieldType"] = "CheckBox" includeNonFloodPointDict["fieldName"] = "includePointsBelowAdvisory" includeNonFloodPointDict["label"] = "Include points below advisory" includeNonFloodPointDict["enable"] = True warningThreshCutOff = {} warningThreshCutOff["fieldType"] = "IntegerSpinner" warningThreshCutOff["fieldName"] = "warningThreshold" warningThreshCutOff["label"] = "Watch/Warning Cutoff Time (hours)" warningThreshCutOff["values"] = self._getWarningThreshold() warningThreshCutOff["minValue"] = 1 warningThreshCutOff["maxValue"] = 72
watchBuffer = {} watchBuffer["fieldType"] = "IntegerSpinner" watchBuffer["fieldName"] = "watchBuffer" watchBuffer["label"] = "Flood Watch Stage/Flow Buffer (%)" watchBuffer["values"] = self._getFloodWatchBufferDefault() watchBuffer["minValue"] = 0 watchBuffer["maxValue"] = 50 miscGroup = { "fieldName": "miscGroup", "fieldType":"Group", "label": "Misc. Options", "expandHorizontally": False, "expandVertically": False, "fields": [includeNonFloodPointDict, warningThreshCutOff, watchBuffer] } if chosenPointID is None: fieldDicts = [choiceFieldGroup, miscGroup, runTypeGroup] valueDict = {"forecastType":"ALL", "includePointsBelowAdvisory":False, "warningThreshold": warningThreshCutOff["values"], "watchBuffer":watchBuffer["values"]} self.addForecastPointChoicesDict(fieldDicts, valueDict) else: fieldDicts = [miscGroup, runTypeGroup] valueDict = {"includePointsBelowAdvisory":True, "warningThreshold": warningThreshCutOff["values"], "watchBuffer":watchBuffer["values"]} valueDict[RUN_TYPE_KEY] = RUN_TYPE_VAL_FULL dialogDict["fields"] = fieldDicts dialogDict["valueDict"] = valueDict dialogDict['siteID'] = self.siteId return dialogDict def filterHazardsByType(self, mergedHazardEvents, currentEvents, dMap): ''' Return only hazards of the desired type (e.g. FL.W, FL.A, etc) ''' filterType = dMap.get('forecastType') includeNonFloodPts = dMap.get("includePointsBelowAdvisory") returnEventSet = EventSet(None) # Force only warning hazards. filterType = "Warning" if filterType == 'ALL': if includeNonFloodPts: return mergedHazardEvents else: for hazardEvent in mergedHazardEvents: phenSig = hazardEvent.getHazardType() if phenSig != 'HY.S': returnEventSet.add(hazardEvent) return returnEventSet excludedEvents = [] for hazardEvent in mergedHazardEvents: phenSig = hazardEvent.getHazardType() if phenSig == 'HY.S' and includeNonFloodPts: returnEventSet.add(hazardEvent) continue elif phenSig == 'HY.S': continue if phenSig == 'FL.W' and filterType == 'Warning': returnEventSet.add(hazardEvent) elif phenSig == 'FL.A' and filterType == 'Watch': returnEventSet.add(hazardEvent) elif phenSig == 'FL.Y' and filterType == 'Advisory': returnEventSet.add(hazardEvent) else: excludedEvents.append(hazardEvent) pass # Add any event not matching a filter type above, but that does match a # point ID of one that was included. This is so that all # recommendations for a given point are returned. includedPointIds = [e.get(POINT_ID) for e in returnEventSet] for hazardEvent in excludedEvents: pointId = hazardEvent.get(POINT_ID) if pointId and pointId in includedPointIds: returnEventSet.add(hazardEvent) return returnEventSet def setHazardType(self, hazardEvent, riverForecastPoint): currentStageTime = MISSING_VALUE currentShefObserved = riverForecastPoint.getCurrentObservation() if currentShefObserved is not None: currentStageTime = currentShefObserved.getObsTime() if currentStageTime == MISSING_VALUE: currentStageTime = SimulatedTime.getSystemTime().getMillis() hazEvtStart = currentStageTime/1000 warningTimeThresh = self.getWarningTimeThreshold(hazEvtStart) hazardEvent.setPhenomenon("FL") if self.isWarning(riverForecastPoint, warningTimeThresh): hazardEvent.setSignificance("W") # elif self.isWatch(riverForecastPoint, warningTimeThresh): # hazardEvent.setSignificance("A") # elif self.isAdvisory(riverForecastPoint): # hazardEvent.setSignificance("Y") else: hazardEvent.setPhenomenon("HY") hazardEvent.setSignificance("S")
eventSignificance = hazardEvent.getSignificance() if eventSignificance == "A" or eventSignificance == "Y" or eventSignificance == "S": hazardEvent.set(HazardConstants.RISE_ABOVE, MISSING_VALUE) hazardEvent.set(HazardConstants.CREST, MISSING_VALUE) hazardEvent.set(HazardConstants.FALL_BELOW, MISSING_VALUE) |
Second, we want to disable the ability to preview and issue these product types from a Hazard Information Dialog. This is done by creating a site override file for HazardTypes.py with the contents that follow. Of course if there is already a site version of HazardTypes.py, these entries would need to be merged in.
HazardTypes.py:
HazardTypes = { |
Finally, we want to make it more obvious that it will not be possible to issue hazards of this type from the Hazard Information Dialog. This is done by putting these three hazard types in a different category. The implementation is to create a site override of HazardCategories.py as follows:
HazardCategories.py:
import collections |
With these mods it will still be possible to view hazards of these types from other offices and manage these hazard types in the settings. This is a bit ugly because it will still be possible in the Hazard Information Dialog to go into the “Unissuable” category and select that hazard type and try to preview or issue, which will result in the operation failing and a traceback being thrown; testing reveals that Hazard Services will recover just fine from this exception. One could decide to not add the Unissuable category, but that would sacrifice all ability to manage these hazard types in the settings. A cleaner way to disable issuing a hazard type will require some updates to the core Hazard Services code.
3.3.4 Modifying existing product content. contents ⇑ ↑ ^ ─ + v ↓ ⇓
3.3.4.1 Adding comparison to flood stage. contents ⇑ ↑ ^ ─ + v ↓ ⇓
The example we will use here is to add a notation about how the reported stage compares to the flood stage in the third bullet of a point flood warning segment, e.g. adding the text in green:
* At 5:30 AM Friday the stage was 10.4 feet...or 0.4 feet above flood stage.
There are two different ways to learn that the product part that produces this text is “observedStageBullet”. First one can invoke the product part diagnostics as in section 3.3.
Second one can note that in the product editor, this appears in the edit window labeled Observed Staged Bullet. Because the product part naming syntax is fairly standard, with experience one can learn that this means the part name would most likely be “observedStageBullet”. Alternatively, one could issue these commands:
> cd /awips2/edex/data/utility/common_static/base/HazardServices/python/textUtilities
> more ProductEditorFields.py
Paging through ProductEditorFields.py, one can note that the product part with a “label” of ‘Observed Stage Bullet’ is "observedStageBullet".
In order to learn which module one needs to override to modify the content of this part, issue these commands (see Introduction for more information about searching code for functionality):
> cd /awips2/edex/data/utility/common_static/base/HazardServices
> touch /tmp/bbb
> find . -name '*py' -exec grep 'def observedStageBullet' '{}' /tmp/bbb \;
This will identify three modules, Legacy_FFA_Formatter.py, Legacy_FLW_FLS_Formatter.py, and SectionLevelMethods.py. As previously mentioned, whenever a product part method appears both in a formatter module and a ‘LevelMethods’ module, it is generally the ‘LevelMethods’ module that creates the content of the part, so this customization needs to be performed by overriding SectionLevelMethods.py. We will show the differences between the base version and the site version in green.
SectionLevelMethods.py:
class SectionLevelMethods(object): def observedStageBullet(self, sectionDict): # There will only be one hazard per section for point hazards eventDict = sectionDict.get('eventDicts')[0] try: numericalFloodCategory = int(eventDict.get('floodCategoryObserved')) except: numericalFloodCategory = -1 if numericalFloodCategory < 0: bulletContent = 'There is no current observed data.' else: observedStageFlow = eventDict.get('observedStage') (stageFlowName, stageFlowValue, stageFlowUnits, combinedValuesUnits) = self._stageFlowValuesUnits(eventDict, observedStageFlow) observedTime = self.fc._getFormattedTime(eventDict.get('observedTime_ms'), timeZones=self.fc.timezones) bulletContent = 'At {0} the {1} was {2}.'.format(observedTime, stageFlowName, combinedValuesUnits) primaryPE = eventDict.get('primaryPE') if self.riverForecastUtils.isPrimaryPeStage(primaryPE) : floodStageFlow = eventDict.get('floodStage') if observedStageFlow > floodStageFlow : dval = observedStageFlow - floodStageFlow ba = 'above' else : dval = floodStageFlow - observedStageFlow ba = 'below' bulletContent += "..or " + "%.1f"%dval + ' feet ' + ba + ' flood stage.' return bulletContent |
3.3.4.2 Adding ice jam specific language. contents ⇑ ↑ ^ ─ + v ↓ ⇓
Here, if the user has selected ice jam as the immediate cause for a point flood warning, we want to add two items of text to the product generated. First we want to add the sentence "The forecast may be affected by ice. " to the end of the forecast bullet. Second, we want to note the location of the ice jam by adding the following text to the warning at an appropriate place:
* An ice jam has formed at near Decatur on the Missouri river.
So let's start by deciding which product parts to add our new text to. Turn on the product part debug information as documented here. Then look at printDebugProductParts.txt after previewing a river flood warning. The forecast bullet is in the part forecastStageBullet. For the sake of argument, let's assume we are going to note the ice jam location in the part called recentActivityBullet.
Now we need to locate where the code is for these product parts (see Introduction for more information about searching code for functionality). One way is to use these commands:
> touch /tmp/bbb
> cd /awips2/edex/data/utility/common_static/base/HazardServices
> find . -name '*py' -exec grep 'def.*recentActivityBullet' '{}' /tmp/bbb \;
> find . -name '*py' -exec grep 'def.*forecastStageBullet' '{}' /tmp/bbb \;
This will identify the modules Legacy_FFA_Formatter.py, Legacy_FLW_FLS_Formatter.py, and
SectionLevelMethods.py for both methods, and so we can override SectionLevelMethods.py to update the content for both parts. So now we want to create a site override file for SectionLevelMethods.py that contains only the forecastStageBullet() and recentActivityBullet() methods. That would look like this:
class SectionLevelMethods(object): def forecastStageBullet(self, sectionDict): # There will only be one hazard per section for point hazards eventDict = sectionDict.get('eventDicts')[0] bulletContent = ForecastStageText().getForecastStageText(eventDict, self.fc.timezones, \ self.fc.productDict.get('issueTime'), \ self.tpc.getVtecAction(sectionDict.get('vtecRecord')), \ sectionDict.get('vtecRecord').get('pil')) return bulletContent def recentActivityBullet(self, sectionDict): bulletContent = '' # There will only be one hazard per section for point hazards eventDict = sectionDict.get('eventDicts')[0] try: numericalFloodCategory = int(eventDict.get('floodCategoryObserved')) except: numericalFloodCategory = -1 if numericalFloodCategory > 0: maxStageFlow = eventDict.get('max24HourObservedStage') stageFlowUnits = eventDict.get('stageFlowUnits') observedTime = self.fc._getFormattedTime(eventDict.get('observedTime_ms'), timeZones=self.fc.timezones).rstrip() (stageFlowName, stageFlowValue, stageFlowUnits, combinedValuesUnits) = self._stageFlowValuesUnits(eventDict, maxStageFlow) bulletContent = 'Recent Activity...The maximum river {0} in the 24 hours ending at {1} was {2}.'\ .format(stageFlowName, observedTime, combinedValuesUnits) return bulletContent |
We need to now learn where we might be able to pick up the river name and the forecast point name within the recentActivityBullet() method. One way is to temporarily add the green code that follows immediately after the line that starts with eventDict =:
eventDict = sectionDict.get('eventDicts')[0] |
If you then preview a point river flood warning, and inspect the contents of /tmp/recentActivityBullet.txt, you will learn that the element "streamName" exists in the dictionary eventDict, which is the name of the river. One can also note that the element "riverPointName" exists in hazard, which is the name of the river forecast point. You can also learn that the element "immediateCause" exists, against which you can test whether the warning is for an ice jam. So the override file that follows can be used to implement these two changes. In green are the changes that note the location of the ice jam, in blue are the changes that add the extra ice jam impact sentence.
class Format(Legacy_Base_Formatter.Format): |
3.3.4.3 Add crest to flood point table. contents ⇑ ↑ ^ ─ + v ↓ ⇓
If one is issuing a point flood warning, there is a checkbox in the “Point Details” tab labeled “Select for flood point table”. If selected, then a table is included in that forecast point segment that shows flood stage, current stage and daily forecast stages for the next three days. The goal of the customization which will be discussed here is to add crest information to this table, e.g., to add the text in green:
. Flood Observed Forecasts 10pm Crest
Location Stg Stg Day/Time Tue Wed Thu Stg Time Date
Platte River
Ashland 20.0 19.4 Mon 10pm MSG MSG MSG 20.8 12am 2/09
As with all such modifications, this starts with previewing a product including the text to be modified and identifying the pertinent product part. The product part that generates this table is “floodPointTable”, which is implemented in multiple formatters and in SegmentLevelMethods.py, which is where the content is generated. So to implement this change one needs to create an override of SegmentLevelMethods.py containing the following (changes/additions in green):
class SegmentLevelMethods(object): |
3.3.4.4. Describe motion of storm causing flooding. contents ⇑ ↑ ^ ─ + v ↓ ⇓
For the vast majority of convective driven flash floods, terrain is the primary forcing of where the area of flooding moves to over time. Because this is true, none of the default settings in the hydro-focused IOC version of Hazard Services have the Storm Track Tool enabled. However, in certain situations with extremely flat terrain, one can have a scenario where the flooding does follow the motion of the storm. To account for this possibility, the Storm Track Tool is available, and it is straightfoward to add this tool to any setting.
This is done using the Edit Default Settings dialog, which is accessed from the console toolbar by making this selection:
SETUP->Manage Settings->Edit/Filter...
Once this dialog is open, choose the Recommenders tab, check Storm Track Tool, and save the setting if desired.
As Hazard Services currently functions, the Storm Track Tool can be used to provide a meaningful first guess of the area under threat for a scenario where the flooding is following a storm. However, there is no language in the resulting FFW text product that describes the storm motion. Here we show how this functionality can be added.
The logic that adds the language is added through the following override to IBW_FFW_FFS_Formatter.py, changes from base in the customary green:
class Format(NWS_Base_Formatter.Format): def getBasisText(self, sectionDict): vtecRecord = sectionDict.get('vtecRecord', {}) phen = vtecRecord.get('phen') sig = vtecRecord.get('sig') eventDicts = sectionDict.get('eventDicts') # FFW_FFS sections will only contain one hazard subType = eventDicts[0].get('subType') hazardType = phen + '.' + sig + '.' + subType basis = self.basisTextModule.getBulletText(hazardType, eventDicts[0], vtecRecord) # Special case for dam and river name, because we may need to look these up # and substitute the framed text based on contents of the dam metadata before # substituteParameters gets a hold of it and does it purely based on a key # in the event attributes. damInfo = {} damOrLeveeName = eventDicts[0].get('damOrLeveeName') if damOrLeveeName : damInfo = self._tpc.damInfoFor(self, damOrLeveeName) if damInfo and not damInfo.get("fromGeneric") : damOrLeveeName = damInfo.get("damName", damOrLeveeName) if HazardConstants.FRAMED_TEXT_DAM_NAME in basis: basis = basis.replace(HazardConstants.FRAMED_TEXT_DAM_NAME, damOrLeveeName) if HazardConstants.FRAMED_TEXT_RIVER_NAME in basis : # replace the riverName with the one from DamMetaData.py riverName = eventDicts[0].get('riverName') if not riverName or riverName.find('|*') > 0 : riverName = damInfo.get('riverName') if riverName: basis = basis.replace(HazardConstants.FRAMED_TEXT_RIVER_NAME, riverName) basis = self._tpc.substituteParameters(eventDicts[0], basis) specLoc = eventDicts[0].get('breachLocation') if not specLoc : specLoc = eventDicts[0].get('iceJamLocation') if not specLoc : specLoc = eventDicts[0].get('glacierLocation') if not specLoc : specLoc = eventDicts[0].get('volcanoName') if specLoc : if specLoc[:3]=='at ' or specLoc[:5]=='near ' or specLoc.find(' of ')>0 : basis = basis.replace('at |* floodLocation *|', specLoc) else : basis = basis.replace('|* floodLocation *|', specLoc) if basis is None : basis = 'Flash Flooding was reported' # Add logic to describe the motion of the storm causing the flooding. stormMotion = eventDicts[0].get('stormMotion') while stormMotion : nCompassPts = 8 # can be 16 cSkip = 16/nCompassPts bearingText = [ 'south', 'south southwest', 'southwest', 'west southwest', 'west', 'west northwest', 'northwest', 'north northwest', 'north', 'north northeast', 'northeast', 'east northeast', 'east', 'east southeast', 'southeast', 'south southeast'] from shapely.geometry import Polygon from MapsDatabaseAccessor import MapsDatabaseAccessor mapsAccessor = MapsDatabaseAccessor() siteID = eventDicts[0].get("siteID") geometry = eventDicts[0].get("geometry") shpFilPolyDict = mapsAccessor.getPolygonDictForTable( \ "warngenloc", siteID, columns=["warngenlev", "lat", "lon"], \ envelopeGeom=geometry) from com.raytheon.uf.common.time import SimulatedTime trackPoints = eventDicts[0].get('trackPoints') locationDicts = eventDicts[0].get('locationDicts', None) if locationDicts: locationsFallBack = "over rural areas of "+\ self._locationsAffectedFallBack(locationDicts) else : locationsFallBack = 'over the aforementioned areas' stormLoc = locationsFallBack if trackPoints and shpFilPolyDict : millis = SimulatedTime.getSystemTime().getMillis() if len(trackPoints)==1 : stormPoint = trackPoints[0].get('point') else : m = len(trackPoints)-1 i = 1 while i<m and long(trackPoints[i+1].get('pointID'))<millis : i += 1 t0 = long(trackPoints[i-1].get('pointID')) t1 = long(trackPoints[i].get('pointID')) wgt0 = (t1-millis)/float(t1-t0) point0 = trackPoints[i-1].get('point') point1 = trackPoints[i].get('point') stormPoint = [ wgt0*point0[0]+(1-wgt0)*point1[0], wgt0*point0[1]+(1-wgt0)*point1[1] ] from DistanceBearing import DistanceBearing from LatLonCoord import LatLonCoord disBearingObj = DistanceBearing(LatLonCoord(stormPoint[1], stormPoint[0])) bestDis = 999999.0 bestInfo = None for wgnId in shpFilPolyDict : shpInfo = shpFilPolyDict[wgnId] wgnPoly = Polygon(shpInfo["geom"]) if not geometry.intersects(wgnPoly) : continue if not bestInfo or \ bestInfo["warngenlev"]>=3 and shpInfo["warngenlev"]<=2 : disBearing = disBearingObj.getDistanceBearing( \ LatLonCoord(shpInfo["lat"],shpInfo["lon"]) ) bestDis = disBearing[0] bestInfo = { "name": wgnId, "warngenlev": shpInfo["warngenlev"], "distance": disBearing[0], "bearing": disBearing[1]} continue if bestInfo["warngenlev"]<=2 and shpInfo["warngenlev"]>=3 : continue disBearing = disBearingObj.getDistanceBearing( \ LatLonCoord(shpInfo["lat"],shpInfo["lon"]) ) if disBearing[0]>bestDis : continue bestDis = disBearing[0] bestInfo = { "name": wgnId, "warngenlev": shpInfo["warngenlev"], "distance": disBearing[0], "bearing": disBearing[1]} # Adaptable parameters are: disUnits overDis nearDis while bestInfo : disUnits = "miles" if disUnits=="nautical miles" : bestDis *= 0.54 else : bestDis /= 1.611 overDis = 3 if bestDis<overDis : stormLoc = "over "+bestInfo["name"] break nearDis = 5 if bestDis<nearDis : stormLoc = "near "+bestInfo["name"] break ffff.write(str(bestDis)+"\n") useBearing = bestInfo["bearing"] b = cSkip*(int(nCompassPts*useBearing/360.0 + 0.5)%nCompassPts) stormLoc = str(int(bestDis+0.5))+" "+disUnits+" "+bearingText[b]+\ " of "+bestInfo["name"] break # Adaptable parameters are: units isStationarySpeed roundSpeedBy units = "miles per hour" if units=="knots" : useSpeed = stormMotion['speed'] else : useSpeed = stormMotion['speed']*1.16 isStationarySpeed = 0.5 if useSpeed <= isStationarySpeed : basis += " The storm producing the flooding is stationary." break roundSpeedBy = 5.0 useSpeed = int( ( useSpeed+(roundSpeedBy/2) ) / roundSpeedBy ) * roundSpeedBy if useSpeed == 0.0 : basis += " The storm producing the flooding is nearly stationary." break useBearing = stormMotion['bearing'] b = cSkip*(int(nCompassPts*useBearing/360.0 + 0.5)%nCompassPts) basis += " The storm producing the flooding is "+stormLoc+", moving "+ \ bearingText[b]+" at "+str(int(useSpeed))+" "+units+"." break return basis |
3.3.4.5 New optional paragraph in overview section of River Flood product.
contents ⇑ ↑ ^ ─ + v ↓ ⇓
Here we demonstrate how to add a new optional paragraph to the overview section of a River Flood product. First, we want to add a megawidget item that allows us to choose whether this paragraph appears. Since this is in the overview section, this needs to be implemented in the MetaData for the staging dialog rather than the MetaData for the Hazard Information Dialog. The staging dialog MetaData for this hazard is in MetaData_FFA_FLW_FLS.py. What identifies this MetaData as being for a staging dialog is that it has text product NNN’s encoded in the filename (e.g. _FLW) rather than phensig’s (e.g. _FL_W). So we begin by providing an override for the execute() method of this class as follows (changes in green):
class MetaData(CommonMetaData.MetaData): # keep it unique, because we could be generating a staging dialog for # multiple product types simultaneously. |
Now we need to preview a River Flood product and identify the product part we want to make our change in. For this example we choose to add our optional text on the front of the ‘groupSummary’ part. So now we need to identify where the code is for this product part, which can be done using these commands (see Introduction for more information about searching code for functionality):
> cd /awips2/edex/data/utility/common_static/base/HazardServices/
> touch /tmp/bbb
> find . -name '*py' -exec grep 'def.*groupSummary' '{}' /tmp/bbb \;
From this we learn that this product part is implemented in the groupSummary() method of multiple formatters and ProductLevelMethods.py, and so the content needs to be overridden using ProductLevelMethods.py. Also, the fact that the associated metadata being modified is in the staging dialog is also a good clue that the change needs to be in ProductLevelMethods.py. Here is the override for ProductLevelMethods.py which adds the new optional paragraph (changes in green):
class ProductLevelMethods(object): |
3.3.4.6 Adding comparison to bankfull stage. contents ⇑ ↑ ^ ─ + v ↓ ⇓
This customization example is unique in that we are going to extend an existing example; specifically Adding comparison to flood stage..
The wording change here is to add a notation about how the reported stage compares to both the flood stage AND the bankfull stage in the third bullet of a point flood warning segment. The wording added from our original example is in blue, and the new wording we are adding here is in green.
* At 1000 PM CST Monday the stage was 30.8 feet...or 4.3 feet above
flood stage and 3.8 feet above bankfull stage.
Note that it is impossible to learn about how to access the bankfull value by inspecting python files; the fact that the riverForecastPoint object has a getBankFull() method available is only knowable through inspecting the Hazard Services java code. The code for the java object that backs the riverForecastPoint object is RiverStationInfo.java, which is posted in Appendix 5. This is being provided so other customizations can leverage the other methods in that class. Even with access to the Hazard Services java code, it is extremely challenging to learn about the relationship between the riverForecastPoint object and RiverStationInfo.java. The original write up including the updates to RiverFloodRecommender.py is available here.
As before we implement this by way of a site override for SectionLevelMethods.py, overriding only the method observedStageBullet(). We will show the changes from the previous example in blue, and the changes that support adding the comparison to bankfull stage in green:
SectionLevelMethods.py:
class SectionLevelMethods(object): def observedStageBullet(self, sectionDict): # There will only be one hazard per section for point hazards eventDict = sectionDict.get('eventDicts')[0] try: numericalFloodCategory = int(eventDict.get('floodCategoryObserved')) except: numericalFloodCategory = -1 if numericalFloodCategory < 0: bulletContent = 'There is no current observed data.' else: observedStageFlow = eventDict.get('observedStage') (stageFlowName, stageFlowValue, stageFlowUnits, combinedValuesUnits) = self._stageFlowValuesUnits(eventDict, observedStageFlow) observedTime = self.fc._getFormattedTime(eventDict.get('observedTime_ms'), timeZones=self.fc.timezones) bulletContent = 'At {0} the {1} was {2}.'.format(observedTime, stageFlowName, combinedValuesUnits) primaryPE = eventDict.get('primaryPE') if self.riverForecastUtils.isPrimaryPeStage(primaryPE) : floodStageFlow = eventDict.get('floodStage') if observedStageFlow > floodStageFlow : dval = observedStageFlow - floodStageFlow ba = 'above' else : dval = floodStageFlow - observedStageFlow ba = 'below' bulletContent += "..or " + "%.1f"%dval + ' feet ' + ba # Now try to add comparison to bankfull. try : bankfull = float(eventDict.get('bankFull')) if observedStageFlow > bankfull : dval = observedStageFlow - bankfull ba2 = 'above' else : dval = bankfull - observedStageFlow ba2 = 'below' if floodStageFlow == bankfull : bulletContent += ' both flood and bankfull stage.' else : bulletContent += ' flood stage and '+ "%.1f"%dval + ' feet ' + ba + ' bankfull stage.' except : bulletContent += ' flood stage.' return bulletContent |
3.3.4.7 Put locations affected into a two column format. contents ⇑ ↑ ^ ─ + v ↓ ⇓
For all areal flood hazards except watches there is a product part labeled Locations Affected. This part typically contains a list of comma delimited locations, mostly towns and cities, which will be impacted by the flood hazard. This customization example shows how to put those locations into a two column format. In this example we demonstrate how one might make this change only impact two hazard types, but one could certainly apply this to all hazard products that contain this part. The override involved is to NWS_Base_Formatter.py, with the changes in the customary green.
class Format(FormatTemplate.Formatter): def createLocationsAffected(self, hazardDict): locations = hazardDict.get('locationsAffected', []) locationsAffected = '' if locations: try : typStr = hazardDict["phen"]+"."+hazardDict["sig"] subTyp = hazardDict.get("subType") if subTyp : typStr += "."+subTyp except : typStr = '' if typStr in ['FF.W.BurnScar', 'FF.W.NonConvective' ] : # Put locations affected in two columns nlnow = True for location in locations : nlnow = not nlnow locStr = location + "..." if nlnow : locationsAffected += locStr+'\n' else : npad = 35-len(locStr) locationsAffected += locStr+npad*" " if not nlnow : locationsAffected += "\n" else : # format this the old way locationsAffected = self._tpc.formatDelimitedList(locations) + '.' else: locationsAffected = 'mainly rural areas of ' locationDicts = hazardDict.get('locationDicts', None) locationDicts = self._tpc.applyGeoAreasToLocationDicts(hazardDict, locationDicts) if locationDicts: locationsAffected += self._locationsAffectedFallBack(locationDicts) else: locationsAffected += 'the aforementioned areas' return locationsAffected |
3.3.5 Validation false alarms. contents ⇑ ↑ ^ ─ + v ↓ ⇓
There will be times when a completely legitimate change in a product format will cause the product validation to improperly indicate that the product is not correctly formatted. Here is a hypothetical example of such a change. Suppose that at site BOU there was a desire to describe Denver County in the first bullet of their unsegmented areal flood hazard products as City and County of Denver. This would mean changing something like this:
* Flash Flood Warning for...
Southwestern Adams County in northeastern Colorado...
Western Arapahoe County in northeastern Colorado...
Southwestern Denver County in northeastern Colorado...
to this:
* Flash Flood Warning for...
Southwestern Adams County in northeastern Colorado...
Western Arapahoe County in northeastern Colorado...
Southwestern City and County of Denver in northeastern Colorado...
This could be accomplished with a simple override to AttributionFirstBulletText.py:
class AttributionFirstBulletText(object):
# First Bullet def getFirstBulletText(self): if self.action == 'CAN': firstBullet = self.firstBullet_CAN() elif self.action == 'EXP': firstBullet = self.firstBullet_EXP() elif self.action == 'UPG': firstBullet = self.firstBullet_UPG() elif self.action == 'NEW': firstBullet = self.firstBullet_NEW() elif self.action == 'EXB': firstBullet = self.firstBullet_EXB() elif self.action == 'EXA': firstBullet = self.firstBullet_EXA() elif self.action == 'EXT': firstBullet = self.firstBullet_EXT() elif self.action == 'CON': firstBullet = self.firstBullet_CON() elif self.action == 'ROU': firstBullet = self.firstBullet_ROU() firstBullet = firstBullet.replace("Denver County", "City and County of Denver") return firstBullet |
The reformatting is easily accomplished, but the result is the following validation false alarm that occurs after Preview:
It is a problem if this dialog comes up every time one issues a legitimate product that includes Denver. If at some point a real problem was identified by the validation, the forecaster would likely just summarily dismiss the validation notice out of force of habit.
This can be dealt with by an override, however, because the validation code has been moved into python. The modularization of the validation code is not optimum right now, and so figuring out how to do this can be challenging. To fix this particular validation false alarm, one needs to override the runQC method in TextSegmentCheck.py. This method is very large, so we will only show the fragment of this method where the change is needed, with the change indicated in the customary green.
: : cpmCounter = 0 for cpm in countyParishMunicipality: if cpm in line: break else: cpmCounter += 1 continue
if (line.find("City and County of Denver")>=0) : countyOrZoneCounter += 1 continue elif (cpmCounter > 0): countyOrZoneCounter += 1 continue else: # ran into a blank line, done insideFirstBullet = False : : |