Hazard Services Start-Up Checklist for New Offices

(Last Update - 8/2/2020 - dlt (patching updates/info)

OVERVIEW - WHAT NEEDS TO BE DONE BEFORE IOC


  1. Complete User/Focal Point TrainingFamiliarize and Train
  2. Complete Basic Configuration
  3. Complete Required Advanced Configuration
  1. Watches, BurnScar, DamCat, MileMarkers, DSSEvents...etc
  1. (Optional) Enable Option to create River Gage Hazards for NonFcstPts
  2. Complete Service Backup Prep Configuration
  1. Run initialization script (hs_export_configuration)
  2. Configure BackupSvc and LocalizationBackupList
  1. HS Overrides Project (Share and Collaborate)
  2. HS Interoperability behavior summary
  3. When ready to have hydro DB copied to HS, email whfs@noaa.gov
  4. In operational mode, check for invalid river flood events (may exist due to past testing).  
  5. Review Support Links
  1. Hazard-Services infolist
  2. nwschat - hazard_services_support group
  3. VLab Hazard Services Community
  4. Focal Point Guide
  1. HS Quick Reference Sheets - (For operations)
  2. Hazard Services Common Practice Workflow (Great for new users to practice HS after taking VLab job sheets)
  3. DAF Resources

AWIPS 2 DR Fix List


HS Patching Scripts (GSL Strategy)

I. Familiarize and Train - (Online Training and Job Sheets) - All Users


  1. Complete User Training for Hazard Services (All Users)
  1. Training available in CLC 
  1. See the "FY20 AWIPS Warning-Related System Readiness - Full" curriculum (for all AWIPS System Readiness Training)
  2. "FY20 AWIPS Warning-Related System Readiness - New" curriculum (for just the FY19 and FY20 training).

  1. Complete Focal Point Training (Focal Point and ITO.  Note other users are welcome to review this information as well)
  1. Training available in CLC 
  1. See the "Hazard Services IOC Focal Point Training" curriculum.

  1. Review Hazard Services Job Sheets (All Users)
  1. Training available on the Hazard Services VLab Site under the Hazard Services Job Sheets pull down menu. 

  1. Understand what Interoperability
  1. Review this shared document to get a better understanding of HS Interoperability

  1. Practice/Familiarize locally in Practice Mode (All Users)
  1. A2 Build 19.3.1 comes with HS Hydro IOC installed.  Users can practice HS in Practice mode.  
  2. Note: Practice mode can be achieved in two ways.
  1. Option #1 - start a single session in practice mode. This is good for testing HS issues only:
  1. (ie /awips2/cave/cave.sh -mode practice)
  1. Option #2 - start an entire workstation in practice mode.  Great for testing RiverPro along with HS.
  1. AWIPS 2 Start Up Menu->Test Mode Control Program

  1. Important - Open Hazard Services in operational mode to check for invalid events
  1. When Hazard Services is first installed, the interoperability process runs in catchup mode to load active events into the registry
  2. It can occasionally pull old river flood (FL.*) events into the registry as active events
  3. It is essential that each site open Hazard Services in operational mode with all hazards filtered in (Hydrology_All settings) and verify that all of the events that show up in the console are valid, active events.
  4. This must be done before transitioning to hazard services for hydro hazard operations
  5. If any invalid events show up, contact the NCF for removal.

II. Basic Configuration - Hazard Services for your WFO (Focal Point/ITO)


  1. Configure visible and possible sites
  1. See Hazard Services VLAB - Set Visible and Possible Sites

  1. Configure Backup Sites
  1. See Hazard Services VLAB - Define Backup Sites

  1. Configure Forecast Initials/Signature
  1. See Hazard Services VLAB - Configure Forecaster Signatures on products

  1. Configure Default settings for your CWA
  1. See Hazard Services VLAB - Change Default Settings

  1. Configure StartUpConfig
  1. Configure settings that are used when Hazard Services is first started
  1. Example: StartUpConfig = { “defaultsettings” : “Hydrology_All” }
  2. Important note about StartupConfig: Do not create any other files in the StartupConfig directory - HS will not start

  1. Configure Inclusion Area for Short Fused Hazards
  1. Option #1: Use Localization Perspective
  1. See Hazard Services VLAB - Configure Inclusion Area for Short Fused Hazards
  2. Note: The inclusion* tags specified in the WarnGen geospatialConfig_*.xml files must be propagated to Hazard Services HazardTypes.py
  1. Optional #2: Use verifyWgnThresholds.csh script ( See Hazard Services Focal Point Users Guide)
  1. cd /awips2/edex/scripts/HazardServices/
  2. ./verifyWgnThresholds.csh XXX (where XXX is your SITE ID)
  3. Use the localization perspective to update Hazard Services -> Hazard Types ->  HazardTypes.py

  1. Configure Extension Area for Short fused Hazards
  1. See Hazard Services VLAB - Configure Extension Area for Short Fused Hazards

  1. Configure Hazard Services alerts
  1. See Hazard Services VLAB - Modifying Alerts

  1. Customize HazardTypesUtils.py
  1. See Focal Point User's Guide - Getting Started Customizing HazardTypes

  1. Examine configuration that Hazard Services uses, but is outside of Hazard Services
  1. Hydro Database: Hazard Services uses a number of configurations from the hydro database that RiverPro also used.  
  2. Here is an active list of hydro database fields to check.

  1.  Review list of temporary overrides for bugs

III. Intermediate/Advanced Configuration - Hazard Services  


  1. Remove watches/advisories from River Flood recommender
  1. See Hazard Services VLAB - Remove the potential to recommend watches and advisories from the river flood recommender

  1. BURN SCAR: Configure Burn Scar Areas
  1. See Hazard Services User Guide - Configure Hazard Services with predefined dams and burn scars
  2. (Jim Ramer Notes)
  3. Option #1: Configure Burn Scar Areas from WarnGen Template
  1. cd /awips2/edex/scripts/HazardServices/
  2. /awips2/python/bin/python parseWarngenTemplate.py -b
  3. The parseWarngenTemplate.py results in new localization files being created at the configured localization level.  No shapefiles are involved.
  1. Option #2: Configure Burn Scar Areas from Shapefile (Need to be polygons)
  1. Stage Burn Scar shapefile into a working directory
  2. Example: cd /localapps/runtime/HazardServices/CO_BurnAreas.shp
  1. Jim’s ingestshapefiles.sh script assumes a default column name of NAME.  
  1. If this is not the case, then you will need to add -a column name below.  
  2. If unsure, you can review the dbf file (ie dam.dbf) and note the attribute name used.  
  3. Example: WFO BOU created a DAM shapefile with the name attribute called DamName  (note case matters)
  4. So I needed to add -c DamName
  5. Run ingestshapefiles.sh by itself to list all  possible options.
  1. dx3<awips>:/awips2/edex/scripts/HazardServices/ingestshapefiles.sh
  1. Usage: ./ingestshapefiles.sh -e -b -u -s /localapps/runtime/HazardServices/CO_BurnAreas.shp -w XXX (Where XXX is the siteid, used for backup sites)
  2. The script will output /tmp/BurnScarMetaData.py.<stamp>. Copy this site to SITE version of CAVE->Hazard Services -> Utilities -> BurnScarMetaData.py.  
  3. Modify BurnScar Recommender
  1. Update SITE File in CAVE->Hazard Services -> Recommenders -> BurnScarFloodRecommender.py

  1. DAMCAT: Configure DamCat
  1. See Hazard Services User Guide - Configure Hazard Services with predefined dams and burn scars
  2. Option #1: Configure predefined DAM from WarnGen Template
  1. cd /awips2/edex/scripts/HazardServices/
  2. /awips2/python/bin/python parseWarngenTemplate.py -d
  3. The parseWarngenTemplate.py results in new localization files being created at the configured localization level.
  1. Option #2: Configure predefined DAM from Shapefile (Need to be polygons)
  1. Stage Dam Cat shapefile into a working directory
  1. Example: cd /localapps/runtime/HazardServices/DAM.shp
  2. Jim’s ingestshapefiles.sh script assumes a default column name of NAME.  
  1. If this is not the case, then you will need to add -a column name below.  
  2. If unsure, you can review the dbf file (ie dam.dbf) and note the attribute name used.  
  3. Example: WFO BOU created a DAM shapefile with the name attribute called DamName  (note case matters)
  4. So I needed to add -c DamName
  1. Use Jim Ramer Script to pull in DAM Shapefiles
  1. dx3<awips>:/awips2/edex/scripts/HazardServices/ingestshapefiles.sh
  1. Usage: ./ingestshapefiles.sh -c columnName (optional. Default is NAME) -e -d -u -s /localapps/runtime/HazardServices/DAM.shp -w XXX (where XXX is your SItE ID, used for backup sites)
  2. The script will output /tmp/DamCatMetaData.py.<stamp>. Copy this site to SITE version of CAVE->Hazard Services -> Utilities -> DamMetaData.py.  
  3. Modify DamBreakFlood Recommender
  4. Update SITE File in CAVE->Hazard Services -> Recommenders -> DamBreakFloodRecommender.py

  1. MILE MARKERS: Convert warnGen mile markers into a form usable by Hazard Services.  
  1. See Hazard Services User Guide section E for detailed information - Convert warnGen information  about mile makers.  Since May 2020, there have been procedures documented in section E for sharing mile marker information with your service backup sites.  The scripts that support this (downloadable from section E) were last updated on June 18, 2020.
  2. Option #1:  This option assumes that you have already warngen configured to use mileMarkers and that the mileMarkers have been imported into the maps database.  
  1. As user awips on dx3, run the commands that follow:
  1. cd /awips2/edex/scripts/HazardServices/
  2. /awips2/python/bin/python parseWarngenTemplate.py -m
  3. This will result in a new localization file being created at the configured localization level in:
  1. Hazard Services -> Event Utilities -> GeoSpatial Config -> PointMarkerConfigs.py
  1. Option #2: Fresh install of mileMarkers.  
  1. Import Mile Markers into warnGen following instructions and example below:
  2. Import MileMarker
  1. cd /awips2/database/sqlScripts/share/sql/maps
  2. ./importMarkersInfo.sh /tmp/i495mm.id mapdata i495mm
  1. Verify
  1. Psql -U awips maps
  2. Select * from mapdata.i25mm;
  1. Rerun Option #1 above.
  1. BOU Mile Marker Configuration Example :
  1. Click here for an example of what BOU (HazardService->GeoSpatial Config-> PointMarkerConfig.py (With DSSEvents enabled)
  1. Code Snip below will automatically check Mile Marker checkbox
  1. SITE CommonMetaData.py (Override getAdditionalLocaltions(self):
  1. values=[]
  2. Above choices.append(newCheckBox) add the following line. values.append(pointConfigName)
  3. Under return section: “values”: values,

  1. WARNGEN DSS EVENTS: Configure warnGen DSS Events to work to HS  ( See  added; there is some information related to this in section  3.0.6 of the Hazard Services Focal Point User’s Guide
  1. If a site has already configured warngen_DSS events in warngen, then adding them to Hazard Services is simple.  
  2. Simply update PointMarkersConfig.py to include dssEvents at the top of this site level override.
  3. BOU WarnGen DSS Events Configuration Example
  1. Click the following link to see an example of how BOU configured this

  1. PointMarker Events: Configure Point Markers to work in HS
  1. This step uses the same PointMarkerConfig used for MileMarkers and DSS Events described above.
  2. For an example and instructions, refer the base file of PointMarkerConfig.py in HazardServices->GeoSpatialConfig  -> PointMarkerConfig.py

IV. ENABLING NON FCST POINTS ON HAZARD SERVICES


  1. Overview: If sites wish to issue river hazards for non fcst points, the river gage needs to be associated with a Fcst Point Group Assignment.  Many sites have already completed this in RiverPro.   If a river gage is NOT associated with a point, Hazard Services will prompt Recommender Completed.  No Recommended Hazards.

  1. Steps to Assign river gage to Fcst Point Group Assignment:
  1. Bring up Hydro Perspective
  2. Identify River Gage
  3. Under the HydroApps menu, bring up Hydro Database Manager
  4. Identify Gage
  5. From the River Gage menu, select River Gage
  6. On the right side, select Forecast Point Group Assignment
  7. Select a Fcst Group (Most likely a group assignment associated with the gage)
  8. Select Apply and Ok

  1. You should now be able to create a Hazard from the modified non fcst point above.

V. SERVICE BACKUP CONFIGURATION


 

  1. Configure Localization Service Backup FIles
  1. NOTE: Backup  services will move files only between sites that are on the same build.  If the sites are not on the same build, the changes will be held in a queue and transferred as soon as the sites are on the same build.
  2. Configure SITE Level backupSvc.xml (Lists the backup sites and other general configuration)
  1. In the Localization Perspective (EDEX->Backup Service):
  1. Copy backupSvc.xml from BASE TO SITE
  2. Change host <name> #1 to dx3-aaa (where aaa is your primary backup site)
  3. Change host <name> #2 to dx3-bbb (where bbb is your secondary backup site)
  4. Save

         

  1. Configure SITE localizationBackupList.txt (Lists which localization files to send to backup sites)
  1. In the Localization Perspective (EDEX->Localization Backup):
  2. Copy localizationBackupList.txt from BASE TO SITE
  3. Adjust as needed, although most offices will be able to use the standard settings listed below.
  4. CONFIGURED settings are included, but this functionality really does not work, since CONFIGURED files are NOT touched via Localization.
  1. For IOC:
  1. SITE:common_static/HazardServices/*
  2. CONFIGURED:common_static/HazardServices/*  
  3. NOTE: 19.3.3+ will keep the following required GFE CONFIGURED files
  1. CONFIGURED:cave_static/gfe/userPython/textUtilities/DefaultAreaDictionary.py and DefaultCityLocation.py
  1. For Post IOC: Hopefully we will be able to include GFE SITE level files with BackupServices
  1. SITE:common_static/HazardServices/*
  2. SITE:cave_static/gfe/userPython/textUtilities
  3. CONFIGURED:common_static/HazardServices/*  
  4. CONFIGURED:cave_static/gfe/userPython/textUtilities

  1. Run Hazard Services Initialization Script
  1. NOTE: In order for Backup Services to work correctly, your backup office needs to be on A2 19.3.1.d
  2. Run the PUSH SCRIPT (A2 19.3.1+) Run an initialization script that pushes your SITE’s Hazard Services configuration files to your backup sites listed in backupSites.xml:
  1. ssh awips@dx3
  2. cd /awips2/edex/scripts/HazardServices/ServiceBackup/scripts/
  3. ./hs_export_configuration -s <SID> -c -g -p
  1. (NOTE: SID is the site that is being exported, i.e. your very own WFO)

  1. Have the WHFS group sync your backups hydro database
  1. The WHFS group has a script to make sure all recently updated hydro data from your backup site is synced with your site

  1. Be sure to include burnScar, damcat, and mileMarker data from your backup office.  If referenced from mapdata database, you will need to make sure this data has been imported into maps.mapdata.

Areas of Typical Customization - A Closer Look  


Overview: There are many areas for sites to customize.  Below are a number of key areas focal points will want to pay particular attention to.  The baseline files for most of these files are sufficient for sites initial functional use of Hazard Services

HazardTypesUtils.py

Class HazardTypesUtils:

def countyExtension(self)

       return (15,1)

def zoneExtension(self)

       return (15,1)

def countyExtension(self)

       return (15,1)

HazardTypes.py

From HazardTypeUtils import HazardTypesUtils

HOURS = 3600000

MINUTEs = 60000

HazardTypes = {

           ‘FF.W.Convective’ : {

                ‘inclusionFraction’: 0.02,

                ‘inclusionAreaInSqKm’: 25 # MUST be a fraction, NOT Percent

                ‘defaultDuration’: 60 * MINUTES,  # 60 * 60000

                ‘durationIncrement’: 15,

                }

        }

HazardCategories.py

HazardCategories = {

 "Hydrology": ["_override_remove_list_",  ("FF", "W", "BurnScar")],

}

Note the _override_remove_list_ control...

HazardMetaData.py

MetaData_XX_Y.py

CallsToActionAndImpacts.py

 

         Example to override CTA verbage:        

class CallsToActionAndImpacts(object):

    ################################################################

    # Copy CTA definition from BASE that you wish to modify wording.  

    ################################################################

    def ctaFlashFloodWatchMeans(self):

        return {"identifier": "FlashFloodWatch",

                "displayString": "Flash Flood Watch means",

                "productString": '''A Flash Flood Watch means rapidly rising water or flooding

                is possible within the watch area.\n\nIf you are in the watch area, keep informed,

                and be ready for quick action if flash flooding is observed or if a warning is issued.'''}

   

    #################################################################

    # Example to override cta verbage (ie. productString)  Each cta in base

    # is referenced by the identifier in the associated Hazard Meta Data files

    #################################################################    

    def ctaActQuickly(self):

        return {"identifier": "actQuicklyCTA",

                "displayString": "Act Quickly",

                "productString":

                "Move to higher ground now. Act quickly to protect your life."}

HazardAlertsConfig.py

HazardAlertsConfig = {
    "eventExpiration": {
        "configuration": [
            "_override_by_key_description_",        
            {
                "description": "Convective Flash Flood Warning",
                "criteria": [
                    "_override_by_key_name_",
                    {"name": "1st warning",
                     "color": {"red": 255, "green": 199, "blue": 0, "alpha": 255},
                    },
                    {"name": "2nd warning",
                     "color": {"red": 255, "green": 0, "blue": 255, "alpha": 255},
                    },
                ]
            },
        ]
    }          

}

Example Override of how to have mileMarkers and dssEvents pre-selected (checked) by default.

class MetaData(object):

    #############################################################

    # Override: This override will auto select the two listed

    # values in the Additional Location settings.    DLT  

    #############################################################

    def getAdditionalLocations(self):

        pointConfigs = self.bridge.getPointConfigs()

        choices = []

        for pointConfigName in pointConfigs:

            pointConfig = pointConfigs.get(pointConfigName)

            newCheckbox = {

                        "identifier" : pointConfigName,

                        "displayString" : pointConfig.get('pointSpecificLabel')

                        }

            if(newCheckbox not in choices):

                choices.append(newCheckbox)

        return {

            "fieldType":"CheckBoxes",

            "fieldName": "additionalLocations",

            "showAllNoneButtons" : False,

            "label": "Additional Locations:",

            "choices": choices,

            "lines": len(choices),    

            ########################################################

            # The line below will auto check both identifiers

            # The indentifiers are listed in the PointMarkersConfig

            #########################################################

            "values" : ["mileMarkers", "dssEvents"]        

            }

Example Override of how to have a CallToAction identifier selected by default.

######################################################

# Updated 01/30/2020 DLH

# CTA OVERRIDE FOR FFW - EXAMPLE FOR OTHERS

######################################################

class MetaData(CommonMetaData.MetaData):

    ##################################################

    # TAG: Done to remove some source choices.

    ##################################################

    def getSourceChoices(self):

        return [

                self.dopplerSource(),

                self.dopplerGaugesSource(),

                self.trainedSpottersSource(),

                self.publicSource(),

                self.localLawEnforcementSource(),

                self.emergencyManagementSource(),

                ############################################

                # Comment out to turn off sources you do not want

                ############################################

                #self.satelliteSource(),

                #self.satelliteGaugesSource(),

                self.gaugesSource(),

                ]

    ###############################################################    

    # TAG: To remove ctaImpact choices, simply comment out below

    ###############################################################

    def getCTA_Choices(self):

        rval = []

        if self.generalizedKeyLookup("ibwType") == 'catastrophic':

            rval.append(self.ctaImpact.ctaFFWEmergency())

           

        rval += [      

            self.ctaImpact.ctaTurnAround(),

            self.ctaImpact.ctaActQuickly(),

            self.ctaImpact.ctaChildSafety(),

            self.ctaImpact.ctaNightTime(),

            self.ctaImpact.ctaUrbanFlooding(),

            self.ctaImpact.ctaRuralFlooding(),

            self.ctaImpact.ctaStayAway(),

            self.ctaImpact.ctaLowSpots(),

            self.ctaImpact.ctaArroyos(),

            self.ctaImpact.ctaBurnAreas(),

            self.ctaImpact.ctaCamperSafety(),

            self.ctaImpact.ctaReportFlooding(),

            self.ctaImpact.ctaFlashFloodWarningMeans(),

            ]

        return rval

   

    #################################################################

    # TAG: Make sure the choice to Act Quickly CTA selected in the GUI.

    # To add additional tags, simply add in the values as noted by

    # example.    #################################################################

    def getCTAs(self, values=None):

        pageFields = {

             "fieldType":"CheckBoxes",

             "fieldName": "cta",

             "showAllNoneButtons" : False,

             "useNewValueOnRefresh": True,

             "choices": self.getCTA_Choices(),

             "values" : [ "actQuicklyCTA" ]

             #"values" : [ "actQuicklyCTA", "stayAwayCTA" ]

         }

        if values is not None:

            pageFields["values"] = values

        return {

               "fieldType": "ExpandBar",

               "fieldName": "CTABar",

               "expandHorizontally": True,

               "pages": [

                            {

                             "pageName": "Calls to Action",

                             "pageFields": [pageFields]

                            }

                         ],

                "expandedPages": ["Calls to Action"]

                }

Understanding Hazard Services use of existing GFE configuration information

the paths to these files after a standard A-II installation (LLL stands for a site ID):

Recommenders (Information taken from Hazard Services Focal Point Guide - Customization

Overview:

        River Flood

        Flash Flood

        Dam Break Flood Recommender

        BurnScar Flood Recommender

Megawidgets - Customizing User Dialogs

VIII. Hazard Services Overrides Project

Overview:  Several folks worked with the VLAB team to enable a process that allows sites to upload their HS Site Level Overrides to VLAB.  The goal is for each office to have its own repository branch that is available to all offices.   The entire process can take place in AWIPS on any workstation that has access to VLAB.   This will allow HS focal points to share/collaborate code snips.  

      The process leverages GIT and may be a little convoluted for new users.  Once authenticated and an initial push of content has occurred, updating future updates is rather simple.   In the notes, I had a suggestion to add a basic text document that identifies unique overrides for your site.   It may be worth adding a separate, searchable WIKI for that as well.

      The instructions can be found here and should be done by a SITE Focal PT in AWIPS.

IX. Example - Creating a ESF Prob Product for monthly Probabilistic Hazard Outlook Steps

X. DAF RESOURCES