The following pseudocode outlines a procedure you can follow using the data lineage API to fetch data lineage links from all available regions.
- Input setup:
linksToFetch
= 5 (number of links to fetch from all regions)regionalPageToken
= "" (last received regional page token used for pagination)- Create:
- list of all regions to fan-out called
regions
. - list of unreachable regions called
unreachable
. - an empty list of found links called
foundLinks
. - an empty variable
nextPageToken
andnextRegionalPageToken
. - an empty variable
lastRegion
. - an empty variable
lastRegionLinksCount
.
- list of all regions to fan-out called
- Sort
regions
alphabetically. - Parse
regionalPageToken
- If it is empty then continue to step 4th
- If it is not empty then
- parse
regionalPageToken
by splitting it by first dot into two segments - filter out regions by taking only the ones that are after (in alphabetical order) the first segment from parsed
regionalPageToken
. - store the second segment aside to use it in all projects.locations.searchLinks calls.
- parse
- Make a call to projects.locations.searchLinks
in each region in the
regions
list in parallel. - Wait for all requests to complete.
- Filter out successful responses and store failed region names in the
unreachable
list. - For each of the responses (starting with first region in alphabetical order)
- if no links were returned along with a non empty page token
- store region name in
lastRegion
. - save received pageToken in
nextPageToken
. - ignore the remaining responses.
- store region name in
- otherwise
- store region name in
lastRegion
. - save received links in
foundLinks
(up tolinksToFetch
). - save received pageToken in
nextPageToken
. - store number of taken links from the response in
lastRegionLinksCount
. - if
foundLinks
is less thanlinksToFetch
- continue the pseudocode with the next received response in the alphabetical order.
- if
foundLinks
equalslinksToFetch
- make another request to projects.locations.searchLinks
endpoint in the same region but set the
pageSize
parameter to the value oflastRegionLinksCount
. - store received pageToken in
nextPageToken
. - continue to step 8
- make another request to projects.locations.searchLinks
endpoint in the same region but set the
- store region name in
- if no links were returned along with a non empty page token
- Prepend nextPageToken with the
lastRegion
(i.e. "[region].[receivedNextPageToken]" to createnextRegionalPageToken
to handle pagination in consecutive calls.