Federator and routing inconsistent

[helle-pedersen]

Hi,

another issue from the student lab work on obspy this morning: inconsistency between EIDA routing and federator. Don't know if it is a routing problem, a federator problem or a NOA problem.
As we tried only networks beginning in C, this problem might existe for other networks?
I just ran it again now (times below).

Cheers
Helle

Describe the problem
The students noticed that EIDA routing and federator didn't give the same number of networks. The difference is the CQ network that appears with the routing but not with the federator.

To Reproduce
With EIDA Routing client:
client = RoutingClient("eida-routing")
inventory = client.get_stations(network="C*")
print(inventory)
Gives networks C4, CA, CH, CK, CL, CP, CQ, CR (2x), CX, CZ
Inventory created at 2025-10-13T13:02:06.880598Z
Created by: ObsPy 1.4.2
https://www.obspy.org/
Sending institution: ObsPy FDSN Routing 1.4.2,RESIF-SI,SeisComP (GFZ,ICGC,NOA,ODC (6.8.4r3b2),RESIF-DC,SED)
Contains:
Networks (11):
C4, CA, CH, CK, CL, CP, CQ, CR (2x), CX, CZ
Stations (613):

---

With Federator
client = Client("EIDA")
inventory = client.get_stations(network="C*")
C4, CA, CH, CK, CL, CP, CR (2x), CX, CZ
print(inventory)
Inventory created at 2025-10-13T07:08:19.472273Z
Sending institution: EIDA-Federator (EIDA)
Contains:
Networks (10):
C4, CA, CH, CK, CL, CP, CR (2x), CX, CZ
Stations (591):

Client
Obspy

Add any other context about the problem here.

[petrrr]

@sheimers and @raciner

This issue is already a bit dated.
Could you have a look if this problem still persists and maybe solve it.

Some quick tests provide a somewhat mixed picture.

For the following query

• https://federator.orfeus-eu.org/fdsnws/station/1/query?level=network&network=CQ

I get a 504 Gateway Time-out. But this might be unrelated to this initial problem addressed here.

This one works for me:

• https://federator.orfeus-eu.org/fdsnws/station/1/query?level=station&network=CQ"

For

• https://federator.orfeus-eu.org/fdsnws/station/1/query?level=station&network=C*

I get 9 networks, the following networks: C4, CA, CH, CK, CL, CP, CQ, CX, CZ.

Which do not exactly match the report by @helle-pedersen. CQ is there, but CR is missing.

[helle-pedersen]

Thanks Peter for following up on this. It is interesting that the renewing the test gives yet another result.

I would suggest to use this simple example to figure out if there is some consistency issue, somewhere between federator, node, or the routing) that we could then systematically check for in the future. This could be part of future developments for EIDA i terms of consistency checks.

Good luck with the investigation!

Helle

[kaestli]

I see now that when asking https://federator.orfeus-eu.org/fdsnws/station/1/query?level=station&network=C* ,
i get (shortened):

<FDSNStationXML schemaVersion="1.1">
<Source>EIDA-Federator</Source>
<Sender>EIDA</Sender>
<Created>2025-12-11T10:55:07.060936</Created>
<Network code="CP" startDate="2018-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CK" startDate="1980-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CQ" startDate="2013-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="C4" startDate="2017-02-15T00:00:00" restrictedStatus="open"></Network>
<Network code="CA" startDate="1980-01-01T00:00:00" restrictedStatus="open"></Network>
<Network code="CH" startDate="1980-01-01T00:00:00" restrictedStatus="open"></Network>
<Network code="CX" startDate="1980-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CL" alternateCode="CRLNET" startDate="2000-01-01T00:00:00" endDate="2500-12-31T23:59:59" restrictedStatus="open"></Network>
<Network code="CZ" startDate="1973-01-01T00:00:00Z" restrictedStatus="open"></Network>
</FDSNStationXML>

... and I get it fast, i.e. from the frontend cache caching the entire compiled result for the exact URI. This cache lasts 1 hour (i think, Stefan to correct if wrong). - so sb. has asked this request within the last hour, and got an incomplete result (partial request to ODC for CR timing out)

If I submit the following request: https://federator.orfeus-eu.org/fdsnws/station/1/query?level=station&network=C*&startafter=1902-01-01T00:00:00
, i force a frontend cache miss (as nobody will have asked fort that exact startafter time within the last hour). Now the federator is expected to go back to the end nodes and collect the information with the new formal restriction. This takes a few seconds more and returns ... :

<FDSNStationXML schemaVersion="1.1">
<Source>EIDA-Federator</Source>
<Sender>EIDA</Sender>
<Created>2025-12-11T17:22:22.505906</Created>
<Network code="CP" startDate="2018-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CK" startDate="1980-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CQ" startDate="2013-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CA" startDate="1980-01-01T00:00:00" restrictedStatus="open"></Network>
<Network code="CR" startDate="2021-01-11T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CH" startDate="1980-01-01T00:00:00" restrictedStatus="open"></Network>
<Network code="C4" startDate="2017-02-15T00:00:00" restrictedStatus="open"></Network>
<Network code="CX" startDate="1980-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CZ" startDate="1973-01-01T00:00:00Z" restrictedStatus="open"></Network>
<Network code="CL" alternateCode="CRLNET" startDate="2000-01-01T00:00:00" endDate="2500-12-31T23:59:59" restrictedStatus="open"></Network>
</FDSNStationXML>

CR included.
So ODC, from the point of view of the federator, was overloaded, timing out, or not reachable when the first, cache-filling request was created. Now, it is not any more (and, as we know, the fdsn standard does not allow to describe partial failure).

Looking at the EIDA availability report @szgrf.bgr.de, we see that ODC is, from the point of view of the federator, unavailable for some 7% of the requests. We will further investigate the cause (whether we use too many parallel connections, or whether those used just cannot cope with the load).

[petrrr]

I understand this is no related to any specific node o network. Rather the federator would miss some random networks if does not manage to obtain the information from the node in time.

I changed the title to account for this.

[petrrr]

@kaestli Not sure I fully why caching is a problem here.

I understand that you might want to case valid replies (status 200) or no data (status 204) but timeouts or other error condition should not be cached, right?

[kaestli]

It is not a priori a problem - it has just an impact on diagnosing the problem.

The federator has, for station requests, two caching levels:

• a frontend cache, just caching the result for a customer request (GET) url for the next hour.
• a backend cache, storing the results of canonized end node requests for the next 12 hours, in order not to bother the end node again if a different request requires the same station information.

Now, if at time x, you (or somebody else) submit a request like https://federator.orfeus-eu.org/fdsnws/station/1/query?level=station&network=C* , this may trigger end node requests for stations of network Ca to endnode A and Cb to end node B. If at the moment of this request, Endnode B answers with 503, the backend will cache only the station contributions of Ca stations. It will assemble a (incomplete) response containing only the stations of network Ca, and return it to you with a status http 200. It will ALSO populate the frontend cache for this specific request for the next hour
So, if at x+5min, I (or somebody else) submit exactly the same request https://federator.orfeus-eu.org/fdsnws/station/1/query?level=station&network=C* again, I will see the result as it was compiled for you, 5 minutes ago.
By adding "&startafter=1902-01-01T00:00:00" (or any other timestamp that hardly anybody used exactly the same way in the last hour) to the user request, I (for analysis reasons) forced bypassing (missing) that frontend cache and urging to assemble the same intended information again from scratch (i.e.: from end nodes). If meanwhile, endnode B has recovered, I would receive a more complete result even before the frontend cache has expired.
This is what i tried to explain above how i confirmed that at the point in time of the test, the issue at ODC was already gone.