This article covers the NTRIP Server sections of the automatic error reporting in greater detail. Instructions regarding how to set the trigger thresholds for the needs for your own Caster are provided below. The SNIP NTRIP Caster provides a range of automatic error reporting mechanisms to alert the Caster Owner/Operator when various detrimental operating conditions occur. A higher level overview of these functional sections is provided in this article.
Overview
The logic involved in this section deals primarily with detecting bad NTRIP Server connections attempts (PUSH_in connections) and reporting these (by eMail) to both the NTRIP operator. This complements other sections that deal with SNIP itself or NTRIP Client connections. In the case of closed Casters using the Reservation system, the email of the registration owner is used for sending reports as well. Informing the Base Station owner provides a means to alert the remote device operators with details of the problem at any hour of the day without operator intervention and at the moment the event occurs.
This section of the dialog consists of the three related logic areas highlighted below.
Each functional area can be enabled / disabled by checking / unchecking the associated check box. Each SetUp button brings up a control dialog where the threshold behavior of that function can be set. For this group of functions the triggering moment is related to when the NTRIP Server connection occurs. Every time an NTRIP Server connects to the Caster, the logs are checked for prior connections and any limits which have been exceeded. This may result in sending a warning eMail out. The Run button causes a summary report to be created. The Run Now button (at the bottom of the dialog) can be used to process all the active functions and send a report.
Each functional area is discussed in turn.
Base Station Off-Line Monitoring
This area deals with the detecting and reporting when a Base Station is offline. Only the Base Stations which are listed are tracked and reported on. If a listed Base Station has NEVER been present since SNIP started, it will not be tracked. Only listed station which have been active in the past (since the last restart) are tracked, this prevents false positive reports, in particular during the initial start up phase of operations.
The Monitored Base Station dialog is similar but slightly more complex than many of the other error reporting threshold dialogs.
Here you can create a list of all the Base Stations that are to be tracked and reported when offline. Not every Base Station requires this, and it can be counter productive to add more than are wanted.
The dialog also has the normal inputs to a) control how often the event must occur (in this case the time interval during which the Base must be offline, in seconds) and b) control if the operator is CCed in the eMail (applicable only to PUSH-In reservation uses).
The value selected for time offline before triggering is a judgement call. A range from 15 to 7200 seconds (2 hours) is supported. In the above example, a time of 10 minutes (600 seconds) was used. Very short times (~15 seconds) reliably report what occurred, but often the NTRIP protocol itself has reconnected and resumed before the operator can react to such events. Very long times (>30 minutes to 2 hours) can be advantageous if the Base Stations have unreliable internet connections and the operator only wishes to be informed of more major events. The default time is 300 seconds (5 minutes).
A drop-down combo box provides a list of all active MountPts from which the operator can select which is to be added. The usual hints showing additional content details about that mountPt are provided for the entries. The MountPts to be tracked are added by selecting them from the list. Once added, they are shown in the dialog. If over 50 mountPts have been selected, the visual list is truncated (you can use the Print command to see the members).
In addition, the following additional controls provide the ability to create and manage the list of Base Stations (mountPts) which is to be tracked.
Clear All
Pressing this button removes all the list members, producing an empty list. This has the same effect as unchecking the check box for this function on the main dialog.
Add All
Pressing this button adds every member of the combo box to the list.
Unless all of your Base Stations connections are unusually reliable, or you have set the off-line period to be large (>1hr), or you are simply monitoring things for debug, such a list is often not suitable. This can result in many eMails being sent if the off-line trigger time is set to be short.
Pressing this button lists every mountPt that is to be tracked to the console list for further review.
Load All Push-In Reservations
This checkbox controls how the combo list is loaded. When unchecked only the active mountPt are loaded (regardless of the type of connection they come from). When checked, all active mountPts are loaded as well as all mountPts in the PUSH_In reservations list. This allows selecting Base Stations which connect by way of a PUSH_In (NTRIP Server) connection when they are not present at that moment in time.
Recall that before any tracked mountPts is reported as offline it must first be present for a short period of time. This prevents false triggering on any tracked mountPts that have never connected to your SNIP Caster.
eMail on data Resume
This checkbox controls the sending of an eMail when the data has resumed. When unchecked no eMail is sent, when checked an eMail is sent when the tracked Base Station data stream resumes and has been declared stable again.
Push_In Connection Issues
This area deals with the most common NTRIP Server connection problems (PUSH_In connections). Issues such as using the wrong password, capitalization problems, NTRIP Revision (Rev1 vs Rev2), missing or incorrect mountPts, and violating various User Account restrictions are all handled here. Ill-formed connections attempts are also reported if they use the POST or SOURCE keywords defined in NTRIP.
The setup dialog allows you to set the number of bad connections which must occur before eMail is sent (in the spin box). You can also set how often another email wold be sent to any connection which continues to make this error (in the combo drop down). You can also CC the operator (or not) on these emails.
Base Station IP Blocked Events
This area deals with sending an eMail when the offending IP has just been blocked. The actual thresholds for blocking are set in the IP blocking dialog. Note the NTRIP Server multiplier control on the IP blocking dialog as well.
The setup dialog does not allow you to set the number of bad connections which must occur before eMail is sent. This value is set in the IP blocking dialog. The only option provided is that you can CC the operator (or not) on these emails.
Typical Examples of use
Here we provide two common NTRIP Client connection problems as examples of use.
- An NTRIP Server Connection (a Base Station) with a bad user password
- An Base Station which has been off-line longer then the allowed time threshold
NTRIP Server Connection (a Base) with a bad user password
NTRIP Server Connection (a Base) with a bad user password is similar to the example given when an NTRIP Client connects with bad credentials. [See the example NTRIP Client Connection with a bad user password ] Based on the current multiplier set in the IP Blocking Dialog, the owner of any incorrect connecting Base Station (or the operator) may be alerted to the bad connection very quickly. As a general design rule, SNIP presumes Base Station devices are better behaved than Client devices. This is effectively enforced by the ratio set in the IP blocking dialog. Once correctly set up, this sort of event should not occur. The settings used here were:
image of how was set here, to be supplied
The first step in this event was when the connecting base itself incorrectly connected for the Nth time reaching the trigger threshold. The console entry for this looks like:
console here, to be supplied
The report eMail which was sent looks like this:
offline email report here, to be supplied
An offline Base Station report
Here we make use of the base “AZU1” – a Base Station sending legacy RTCM3 close to our California offices nicknamed “old unfaithful” for its tendency to briefly go off-line two or more times a day at different times. The reason for this odd behavior is a lack of real time data capacity in the local network which operates it. Twice a day this station will go off-line for all the real time users so it can provide an FTP transfer of the prior ~12 hours of raw data to its owner. This ensures that the UNAVCO RINEX file sets are complete. A few minutes thereafter, it then returns to an operational state. The precise time at which this occurs varies depending on the time required by similar downloads collected from nearby stations. If it happens to be on-line when you are reading this, you can get a quick status summary from one of our SNIP Casters at this link.
For this example we set the time threshold to be fairly short, 180 seconds. The value selected is always a compromise between a short value (tell me about any problems right NOW!) and a longer value to avoid false positives that often correct themselves before any action can be taken to correct things. When this Base goes offline the SNIP console log displays the normal warning messages.
And the email which was sent to the SNIP operator contains the below. Note that the state of any other Base Stations that are also big monitored can also be reported here.
This is an automatic report from the XXXNode NTRIP Caster machine at 192.168.2.105:2101
WARNING:
Issued at Wed September 8 2021 at 16:15:17 Pacific Daylight Time
The remote-relay mountPt AZU1 has been offline for the past 03:05 MIN:SEC, have tried to reconnect 20/15 times.
In the case of this Base Station, it typically resumes sending which results in the following ‘up again’ eMail being sent (if the eMail on data Resume checkbox is checked) after a short period of stable operation was observed.
This is an automatic report from the XXXNode NTRIP Caster machine at 192.168.2.105:2101 NOTICE: Issued at Wed September 8 2021 at 16:29:23 Pacific Daylight Time The Base Station AZU1 has resumed sending data to the Caster. The Base Station was first declared offline at: Wed September 8 2021 at 16:15:17 Pacific Daylight Time The outage lasted for about: 14:06 MIN:SEC When the Base is operational, you can review the current status with the below link: AZU1 Current Status (link removed) If the Base Station goes offline for more than 180 seconds, you will be sent a new alert email.