Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -178,10 +178,12 @@ def poll_func(rect: LatLngRect) -> bool:
poll_func,
)

def _is_area_too_large(self, rect: s2sphere.LatLngRect) -> bool:
return geo.get_latlngrect_diagonal_km(rect) > self._rid_version.max_diagonal_km

def _fetch_flights_from_dss(self, rect: LatLngRect) -> dict[str, TelemetryMapping]:
# We grab all flights from the SPs (which we know how to reach by first querying the DSS).
# This is authenticated and is expected to succeed
# TODO: Add the following requests to the documentation. Possibly split it as a test step.
sp_observation = rid.all_flights(
rect,
include_recent_positions=True,
Expand All @@ -191,14 +193,27 @@ def _fetch_flights_from_dss(self, rect: LatLngRect) -> dict[str, TelemetryMappin
dss_participant_id=self._dss.participant_id,
)

self.record_queries(sp_observation.queries)

with self.check("Initial queries", [self._dss.participant_id]) as check:
if not sp_observation.success and not self._is_area_too_large(rect):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you noticed in your PR comment, rid.all_flights doesn't just perform an ISA query; it performs an ISA query (potentially) followed by a bunch of queries for flights and flight details from individual USSs. The DSS failing to return ISAs is a violation of different requirements than the core USS failing to respond correctly to a flights query or a flight details query. Probably part of #1386 involves resolving the TODO just above this section.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think the todo is resolved, however the name is wrong (and probably wrong for existing ISA query as well, since they do the same).

Do you have a better name in mind?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For misbehaviour I switched to "Initial queries" and improved description based on comment bellow, but I think the point still hold for display data evaluator ones.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remember that test steps are where actions are performed, and then checks (following the actions) determine if any requirements were violated based on the information obtained from those actions. With that in mind, I'm not sure what "Initial queries" would be checking. I think this first check description makes sense:

interuss.f3411.dss_endpoints.SearchISAs requires a USS providing a DSS instance to implement the DSS endpoints of the OpenAPI specification. If uss_qualifier is unable to query the DSS for ISAs, this check will fail.

But, it seems like the thing we're checking is "Did the (single) DSS query to search for ISAs succeed?" I would expect that check to be named something like "Successful ISA search" or "DSS ISAs search succeeded" or "ISA query succeeded" or "ISA query success".

This check description makes less sense to me (additional emphasis added):

interuss.f3411.dss_endpoints.SearchISAs requires a USS providing a DSS instance to implement the DSS endpoints of the OpenAPI specification. Then, in order to properly test whether the SP handles authentication correctly, after identifying the SP contact information via its ISA in the DSS, this step will first attempt to do a flights request with the proper credentials to confirm that the requested data is indeed available to any authorized query. If uss_qualifier is unable to query the DSS for ISAs or flights, this check will fail.

This seems to be documenting a check (the first sentence) then another, separate test step (second sentence). Actions in a test step belong in the test step documentation (one documentation level higher than check documentation) and then checks should describe only how the information already obtained will be evaluated to determine if a requirement was violated.

So, it seems like there are three actions in this test step (due to rid.all_flights):

  1. Query DSS for ISAs
  2. For each ISA found, query the USS for flights
  3. For each flight found, query the USS for flight details

The InterUSS requirement interuss.f3411.dss_endpoints.SearchISAs applies only to the information obtained from action 1. interuss.f3411.dss_endpoints.SearchISAs doesn't relate at all to the information obtained in actions 2 nor 3 -- instead, presumably requirements NET0710,1 and NET0710,2 apply to actions 2 and 3, respectively (when those queries are expected to succeed). But, these should be two separate checks since the information we're evaluating for NET0710,1 is coming from action 2 and the information we're evaluating for NET0710,2 is coming from action 3 -- the action, information, and relevant requirement are all different between action 2/NET0710,1 and action 3/NET0720,2 so they shouldn't be combined into a single check.

The original problem prompting #1386 was that these two checks AND the ISA query check were all combined into a single check. When that check failed, it was extremely unclear what had actually gone wrong. That's because a check for A && B && C && D failing provides little information as to whether A failed, B failed, and/or C failed. This was especially problematic because the check only documented that D was being checked, so the fact that B was failing was extremely difficult to determine.

check.record_failed(
summary="Could not query ISAs from DSS",
details=f"Query to {self._dss.participant_id}'s DSS at failed: {', '.join(sp_observation.errors)}",
query_timestamps=[
query.request.initiated_at.datetime
for query in sp_observation.queries
if query.request.initiated_at
],
)

mapping_by_injection_id = (
display_data_evaluator.map_fetched_to_injected_flights(
self._injected_flights,
list(sp_observation.uss_flight_queries.values()),
self._query_cache,
)
)
self.record_queries(sp_observation.queries)

return mapping_by_injection_id

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -284,14 +284,29 @@ def evaluate_system_instantaneously(
dss_participant_id=self._dss.participant_id,
)

for q in sp_observation.queries:
self._test_scenario.record_query(q)

with self._test_scenario.check(
"ISA query", [self._dss.participant_id]
) as check:
if not sp_observation.success and not self._is_area_too_large(rect):
check.record_failed(
summary="Could not query ISAs from DSS",
details=f"Query to {self._dss.participant_id}'s DSS at failed: {', '.join(sp_observation.errors)}",
query_timestamps=[
query.request.initiated_at.datetime
for query in sp_observation.queries
if query.request.initiated_at
],
)

# map observed flights to injected flight and attribute participant ID
mapping_by_injection_id = map_fetched_to_injected_flights(
self._injected_flights,
list(sp_observation.uss_flight_queries.values()),
self._query_cache,
)
for q in sp_observation.queries:
self._test_scenario.record_query(q)

# Evaluate observations
self._evaluate_sp_observation(rect, sp_observation, mapping_by_injection_id)
Expand Down Expand Up @@ -325,6 +340,9 @@ def evaluate_system_instantaneously(
# TODO: If bounding rect is smaller than area-too-large threshold, expand slightly above area-too-large threshold and re-observe
self._test_scenario.end_test_step()

def _is_area_too_large(self, rect: s2sphere.LatLngRect) -> bool:
return geo.get_latlngrect_diagonal_km(rect) > self._rid_version.max_diagonal_km

def _evaluate_observation(
self,
observer: RIDSystemObserver,
Expand All @@ -333,10 +351,8 @@ def _evaluate_observation(
query: fetch.Query,
verified_sps: set[str],
) -> None:
diagonal_km = (
rect.lo().get_distance(rect.hi()).degrees * geo.EARTH_CIRCUMFERENCE_KM / 360
)
if diagonal_km > self._rid_version.max_diagonal_km:
diagonal_km = geo.get_latlngrect_diagonal_km(rect)
if self._is_area_too_large(rect):
self._evaluate_area_too_large_observation(
observer, rect, diagonal_km, query
)
Expand Down Expand Up @@ -1070,14 +1086,29 @@ def evaluate_disconnected_flights(
dss_participant_id=self._dss.participant_id,
)

for q in sp_observation.queries:
self._test_scenario.record_query(q)

with self._test_scenario.check(
"ISA query", [self._dss.participant_id]
) as check:
if not sp_observation.success:
check.record_failed(
summary="Could not query ISAs from DSS",
details=f"Query to {self._dss.participant_id}'s DSS at failed: {', '.join(sp_observation.errors)}",
query_timestamps=[
query.request.initiated_at.datetime
for query in sp_observation.queries
if query.request.initiated_at
],
)

# map observed flights to injected flight and attribute participant ID
mapping_by_injection_id = map_fetched_to_injected_flights(
self._injected_flights,
list(sp_observation.uss_flight_queries.values()),
self._query_cache,
)
for q in sp_observation.queries:
self._test_scenario.record_query(q)

# Evaluate observations
self._evaluate_sp_observation(sp_observation, mapping_by_injection_id)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,11 @@ This particular test requires each flight to be uniquely identifiable by its 2D

### Invalid search area test step

This step will attempt to search for flights in a rectangular area with a diagonal greater than [NetMaxDisplayAreaDiagonal] km.
This step will attempt to search for flights in a rectangular area with a diagonal greater than [NetMaxDisplayAreaDiagonal] km. First, the Service Providers with service in the large area will be determined from the DSS (this should succeed), and then each Service Provider will be queried for flights using an unacceptably-large area (this should fail).

#### ⚠️ Initial queries check

**[interuss.f3411.dss_endpoints.SearchISAs](../../../../requirements/interuss/f3411/dss_endpoints.md)** requires a USS providing a DSS instance to implement the DSS endpoints of the OpenAPI specification. If uss_qualifier is unable to query the DSS for ISAs, this check will fail.

#### ⚠️ Area too large check

Expand All @@ -54,6 +58,10 @@ to confirm that the requested data is indeed available to any authorized query.

It then repeats the exact same request without credentials, and expects this to fail.

#### ⚠️ Initial queries check

**[interuss.f3411.dss_endpoints.SearchISAs](../../../../requirements/interuss/f3411/dss_endpoints.md)** requires a USS providing a DSS instance to implement the DSS endpoints of the OpenAPI specification. Then, in order to properly test whether the SP handles authentication correctly, after identifying the SP contact information via its ISA in the DSS, this step will first attempt to do a flights request with the proper credentials to confirm that the requested data is indeed available to any authorized query. If uss_qualifier is unable to query the DSS for ISAs or flights, this check will fail.

#### ⚠️ Missing credentials check

This check ensures that all requests are properly authenticated, as required by **[astm.f3411.v19.NET0210](../../../../requirements/astm/f3411/v19.md)**,
Expand All @@ -63,6 +71,10 @@ and that requests for existing flights that are executed with missing credential

This step is similar to unauthenticated requests, but uses incorrectly-authenticated requests instead.

#### ⚠️ Initial queries check

**[interuss.f3411.dss_endpoints.SearchISAs](../../../../requirements/interuss/f3411/dss_endpoints.md)** requires a USS providing a DSS instance to implement the DSS endpoints of the OpenAPI specification. Then, in order to properly test whether the SP handles authentication correctly, after identifying the SP contact information via its ISA in the DSS, this step will first attempt to do a flights request with the proper credentials to confirm that the requested data is indeed available to any authorized query. If uss_qualifier is unable to query the DSS for ISAs or flights, this check will fail.

#### ⚠️ Invalid credentials check
This check ensures that all requests are properly authenticated, as required by **[astm.f3411.v19.NET0210](../../../../requirements/astm/f3411/v19.md)**,
and that requests for existing flights that are executed with incorrect credentials fail.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,11 @@ A [`DSSInstanceResource`](../../../../resources/astm/f3411/dss.py) is required f

### Invalid search area test step

This step will attempt to search for flights in a rectangular area with a diagonal greater than [NetMaxDisplayAreaDiagonal] km.
This step will attempt to search for flights in a rectangular area with a diagonal greater than [NetMaxDisplayAreaDiagonal] km. First, the Service Providers with service in the large area will be determined from the DSS (this should succeed), and then each Service Provider will be queried for flights using an unacceptably-large area (this should fail).

#### ⚠️ Initial queries check

**[interuss.f3411.dss_endpoints.SearchISAs](../../../../requirements/interuss/f3411/dss_endpoints.md)** requires a USS providing a DSS instance to implement the DSS endpoints of the OpenAPI specification. If uss_qualifier is unable to query the DSS for ISAs, this check will fail.

#### ⚠️ Area too large check

Expand All @@ -41,6 +45,10 @@ to confirm that the requested data is indeed available to any authorized query.

It then repeats the exact same request without credentials, and expects this to fail.

#### ⚠️ Initial queries check

**[interuss.f3411.dss_endpoints.SearchISAs](../../../../requirements/interuss/f3411/dss_endpoints.md)** requires a USS providing a DSS instance to implement the DSS endpoints of the OpenAPI specification. Then, in order to properly test whether the SP handles authentication correctly, after identifying the SP contact information via its ISA in the DSS, this step will first attempt to do a flights request with the proper credentials to confirm that the requested data is indeed available to any authorized query. If uss_qualifier is unable to query the DSS for ISAs or flights, this check will fail.

#### ⚠️ Missing credentials check

This check ensures that all requests are properly authenticated, as required by **[astm.f3411.v22a.NET0210](../../../../requirements/astm/f3411/v22a.md)**,
Expand All @@ -50,6 +58,10 @@ and that requests for existing flights that are executed with missing credential

This step is similar to unauthenticated requests, but uses incorrectly-authenticated requests instead.

#### ⚠️ Initial queries check

**[interuss.f3411.dss_endpoints.SearchISAs](../../../../requirements/interuss/f3411/dss_endpoints.md)** requires a USS providing a DSS instance to implement the DSS endpoints of the OpenAPI specification. Then, in order to properly test whether the SP handles authentication correctly, after identifying the SP contact information via its ISA in the DSS, this step will first attempt to do a flights request with the proper credentials to confirm that the requested data is indeed available to any authorized query. If uss_qualifier is unable to query the DSS for ISAs or flights, this check will fail.

#### ⚠️ Invalid credentials check

This check ensures that all requests are properly authenticated, as required by **[astm.f3411.v22a.NET0210](../../../../requirements/astm/f3411/v22a.md)**,
Expand Down
2 changes: 1 addition & 1 deletion monitoring/uss_qualifier/suites/astm/netrid/f3411_19.md
Original file line number Diff line number Diff line change
Expand Up @@ -561,7 +561,7 @@
<tr>
<td><a href="../../../requirements/interuss/f3411/dss_endpoints.md">SearchISAs</a></td>
<td>Implemented</td>
<td><a href="../../../scenarios/astm/netrid/v19/dss/heavy_traffic_concurrent.md">ASTM NetRID DSS: Concurrent Requests</a><br><a href="../../../scenarios/astm/netrid/v19/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../../scenarios/astm/netrid/v19/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../../scenarios/astm/netrid/v19/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a><br><a href="../../../scenarios/astm/netrid/v19/sp_operator_notify_slow_update.md">ASTM NetRID Service Provider operator notification under slow update rate</a><br><a href="../../../scenarios/astm/netrid/v19/networked_uas_disconnect.md">ASTM NetRID networked UAS disconnection</a><br><a href="../../../scenarios/astm/netrid/v19/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
<td><a href="../../../scenarios/astm/netrid/v19/dss/heavy_traffic_concurrent.md">ASTM NetRID DSS: Concurrent Requests</a><br><a href="../../../scenarios/astm/netrid/v19/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../../scenarios/astm/netrid/v19/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../../scenarios/astm/netrid/v19/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a><br><a href="../../../scenarios/astm/netrid/v19/misbehavior.md">ASTM NetRID SP clients misbehavior handling</a><br><a href="../../../scenarios/astm/netrid/v19/sp_operator_notify_slow_update.md">ASTM NetRID Service Provider operator notification under slow update rate</a><br><a href="../../../scenarios/astm/netrid/v19/networked_uas_disconnect.md">ASTM NetRID networked UAS disconnection</a><br><a href="../../../scenarios/astm/netrid/v19/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
</tr>
<tr>
<td rowspan="1" style="vertical-align:top;"><a href="../../../requirements/interuss/mock_uss/hosted_instance.md">interuss<br>.mock_uss<br>.hosted_instance</a></td>
Expand Down
2 changes: 1 addition & 1 deletion monitoring/uss_qualifier/suites/astm/netrid/f3411_22a.md
Original file line number Diff line number Diff line change
Expand Up @@ -681,7 +681,7 @@
<tr>
<td><a href="../../../requirements/interuss/f3411/dss_endpoints.md">SearchISAs</a></td>
<td>Implemented</td>
<td><a href="../../../scenarios/astm/netrid/v22a/dss/heavy_traffic_concurrent.md">ASTM NetRID DSS: Concurrent Requests</a><br><a href="../../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../../scenarios/astm/netrid/v22a/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a><br><a href="../../../scenarios/astm/netrid/v22a/sp_operator_notify_slow_update.md">ASTM NetRID Service Provider operator notification under slow update rate</a><br><a href="../../../scenarios/astm/netrid/v22a/networked_uas_disconnect.md">ASTM NetRID networked UAS disconnection</a><br><a href="../../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
<td><a href="../../../scenarios/astm/netrid/v22a/dss/heavy_traffic_concurrent.md">ASTM NetRID DSS: Concurrent Requests</a><br><a href="../../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../../scenarios/astm/netrid/v22a/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a><br><a href="../../../scenarios/astm/netrid/v22a/misbehavior.md">ASTM NetRID SP clients misbehavior handling</a><br><a href="../../../scenarios/astm/netrid/v22a/sp_operator_notify_slow_update.md">ASTM NetRID Service Provider operator notification under slow update rate</a><br><a href="../../../scenarios/astm/netrid/v22a/networked_uas_disconnect.md">ASTM NetRID networked UAS disconnection</a><br><a href="../../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
</tr>
<tr>
<td rowspan="1" style="vertical-align:top;"><a href="../../../requirements/interuss/mock_uss/hosted_instance.md">interuss<br>.mock_uss<br>.hosted_instance</a></td>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -673,7 +673,7 @@
<tr>
<td><a href="../../requirements/interuss/f3411/dss_endpoints.md">SearchISAs</a></td>
<td>Implemented</td>
<td><a href="../../scenarios/astm/netrid/v22a/dss/heavy_traffic_concurrent.md">ASTM NetRID DSS: Concurrent Requests</a><br><a href="../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../scenarios/astm/netrid/v22a/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a><br><a href="../../scenarios/astm/netrid/v22a/sp_operator_notify_slow_update.md">ASTM NetRID Service Provider operator notification under slow update rate</a><br><a href="../../scenarios/astm/netrid/v22a/networked_uas_disconnect.md">ASTM NetRID networked UAS disconnection</a><br><a href="../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
<td><a href="../../scenarios/astm/netrid/v22a/dss/heavy_traffic_concurrent.md">ASTM NetRID DSS: Concurrent Requests</a><br><a href="../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../scenarios/astm/netrid/v22a/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a><br><a href="../../scenarios/astm/netrid/v22a/misbehavior.md">ASTM NetRID SP clients misbehavior handling</a><br><a href="../../scenarios/astm/netrid/v22a/sp_operator_notify_slow_update.md">ASTM NetRID Service Provider operator notification under slow update rate</a><br><a href="../../scenarios/astm/netrid/v22a/networked_uas_disconnect.md">ASTM NetRID networked UAS disconnection</a><br><a href="../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
</tr>
<tr>
<td rowspan="1" style="vertical-align:top;"><a href="../../requirements/interuss/mock_uss/hosted_instance.md">interuss<br>.mock_uss<br>.hosted_instance</a></td>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1121,7 +1121,7 @@
<tr>
<td><a href="../../requirements/interuss/f3411/dss_endpoints.md">SearchISAs</a></td>
<td>Implemented</td>
<td><a href="../../scenarios/astm/netrid/v22a/dss/heavy_traffic_concurrent.md">ASTM NetRID DSS: Concurrent Requests</a><br><a href="../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../scenarios/astm/netrid/v22a/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a><br><a href="../../scenarios/astm/netrid/v22a/sp_operator_notify_slow_update.md">ASTM NetRID Service Provider operator notification under slow update rate</a><br><a href="../../scenarios/astm/netrid/v22a/networked_uas_disconnect.md">ASTM NetRID networked UAS disconnection</a><br><a href="../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
<td><a href="../../scenarios/astm/netrid/v22a/dss/heavy_traffic_concurrent.md">ASTM NetRID DSS: Concurrent Requests</a><br><a href="../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../scenarios/astm/netrid/v22a/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a><br><a href="../../scenarios/astm/netrid/v22a/misbehavior.md">ASTM NetRID SP clients misbehavior handling</a><br><a href="../../scenarios/astm/netrid/v22a/sp_operator_notify_slow_update.md">ASTM NetRID Service Provider operator notification under slow update rate</a><br><a href="../../scenarios/astm/netrid/v22a/networked_uas_disconnect.md">ASTM NetRID networked UAS disconnection</a><br><a href="../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
</tr>
<tr>
<td rowspan="1" style="vertical-align:top;"><a href="../../requirements/interuss/f3548/notification_requirements.md">interuss<br>.f3548<br>.notification_requirements</a></td>
Expand Down
Loading