# Vector reads, XCache and WAN latency — what we measured, 2026-09-01

An investigation into why an XCache cold run of the AGC ttbar analysis was
dramatically slower than the same analysis reading direct from the same origin.
It ended without a proven mechanism, but it eliminated a lot and produced two
solid, unrelated results. The eliminations are the most valuable part: most of
them were confidently asserted first and disproved after.

Companion docs: `claude-blockrun-state.md` (the XrdPfc BlockRun work),
`xcache-step1-small-reads.md`, `claude-pgReadV-bootstrap.md`.

## The setup

Workload: `root-project/analysis-grand-challenge`, `analyses/cms-open-data-ttbar`,
`analysis.py -n 1 --scheduler mt --ncores 8 --no-fitting`. RDataFrame, 8 samples
of CMS UL16 NanoAODv9. Client always on `black` (home, ATT fibre).

Origins used, all with the same cache code and `pfc.cschk off`:

| origin | kind | RTT from black |
|---|---|---|
| `xrootd.t2.ucsd.edu:21094` | stock xrootd 5.9.6, ours | 9 ms |
| `cmsdcadisk.fnal.gov:1094` | dCache xrootd door | 59 ms |
| `eospublic.cern.ch:1094` | EOS | 152 ms |
| `ouroboros.cern.ch:21099` | stock xrootd 5.9.7, ours | 152 ms (via ssh tunnel) |

## The two solid results

**1. Block size dominates over-fetch, by a lot.** Same analysis, prefetch off,
verified by exact block accounting (9185 blocks x 1 MB = 9.0 GB, matching disk
usage to the digit):

| pfc.blocksize | data fetched | cold execute |
|---|---|---|
| 4 kB | 823 MB | 247 s |
| 1 MB | 9.3 GB | 554 s |

**11x less data for identical physics.** A synthetic file with few branches and
fat baskets showed only 4.7x; it took real NanoAOD (~1500 branches, small
baskets) to show the true figure.

**2. On a clean path the cache costs almost nothing.** UCSD, 9 ms, direct, plain
xrootd, reproducible to +-1 ms:

| | latency |
|---|---|
| direct client, 1152 kB readv | 20.0 ms |
| direct client, 1408 kB readv | 20.9 ms |
| cache (sends 1408 kB for a 1152 kB request) | 22 ms |

The cache's overhead is ~2 ms, of which ~1 ms is the unavoidable +22.2% from
4 kB block rounding. Warm cache beats direct by 2.2x on the full analysis.

## The unexplained part, at this point

*Resolved further down. Kept as written to show what the evidence looked
like before the cause was found.*

Against FNAL, the cache's vector reads averaged **415 ms** where ROOT's averaged
**178.8 ms**, reproducibly (cold analysis: 247/245/241/240/223 s vs 28 s).
At UCSD the same comparison is 22.0 vs 22.2 ms — no difference at all.

So the penalty appears only at high RTT, and no client-side property explains
it. Everything below was measured and found not to be the cause.

## Hypotheses eliminated

**`xrootd.async` 64 kB segmentation.** `do_ReadV` contains no aio at all —
verified in `XrdXrootdXeq.cc`; only `do_Read` has the async path. Vector reads
are never segmented. `xrootd.async off` and `nocache` both changed nothing
(241 and 240 s vs 247 s baseline). Only 8% of bytes took the plain-read path.

**Cache-internal locking / `m_block_map` contention.** The warm run pushes the
same 666 MB through the same code, same locks, same 8 threads at 57 MB/s.

**Origin request count.** 595 origin readv for 587 client readv — one for one,
no amplification.

**Request shape.** With `pfc.iosize 1M` the cache sends *fewer* elements than
ROOT (33795 vs 39731 for the same 587 requests), same 140 kB max element, and
+22% bytes. Exhaustive check of every request in every run: zero non-monotonic,
zero duplicate offsets, zero overlaps, zero zero-length elements.

**The XrdPosix / XrdCl path.** `XrdPosixFile::ReadV` and
`FileStateHandler::VectorRead` are both straight pass-throughs;
`XrdCl::Utils::SplitChunks` is not on this path. What `ReadOpusCoalescere`
builds is what goes on the wire.

**XrdCl version.** A synthetic 64-chunk 1152 kB readv: 71.4 ms on system 5.9.2,
71.0 ms on our 6.1.x build.

**Protocol, TLS, auth, topology.** Both negotiate protocol 500, both TLS to the
door and neither to the pools, both gsi with the same DN, both use the same 9
pool nodes over 9 connections with matching per-pool request counts. Confirmed
by socket count: 12 established sockets, one per pool host, for both.

**pgReadV.** Never called: `cschk off` takes the ReadV branch. Wire shows 595
kXR_readv, 0 kXR_pgread.

**Server implementation.** Initially concluded "it is the dCache door". Wrong:
EOS shows the same ~2x, and so does a *stock xrootd* at CERN.

**`net.core.rmem_max`.** Looked compelling — 212992 on the UCSD host, and
latency rose in steps of about that size. But `XrdNetSocket::setWindow` is only
called when a window is explicitly configured, which ours is not, so
`SO_RCVBUF` is never set and `rmem_max` never applies. Autotuning governs, with
the `net.ipv4.tcp_rmem` ceiling (6 MB at UCSD, 32 MB on black). Coincidence
fitting.

**`tcp_slow_start_after_idle`.** Set to 1 on both hosts, so plausible. But the
cache's origin connection is idle only 17 ms between readv (p90 22 ms), and only
1% of gaps exceed a 200 ms RTO. It essentially never fires.

**Multiple substreams.** `DefaultSubStreamsPerChannel` is 1 and the cache uses
one TCP connection per origin — but `XRD_SUBSTREAMSPERCHANNEL=4` made no
difference (324/288/332/334 ms). A single response travels one stream.

**A byte cap on the origin vector request.** Implemented, then reverted
unmerged: with one substream, splitting a readv sends the same bytes through the
same connection and window. It cannot help.

**"ROOT stays under a size threshold, we cross it."** Wrong. ROOT's readv sizes
are data-driven and overlap ours: min 75, median 1226, p90 1626, max 2038 kB.
ROOT routinely sends *larger* requests than our 1408 kB.

## What the kernel actually shows

`ss -tin` on the cache's FNAL sockets, mid-transfer, 100+ MB received:

```
rcv_space: 1465120        receive window ~1.46 MB
rcv_ssthresh: 10599918    headroom to 10.6 MB, unused
app_limited               set on every socket
rtt: 54-72 ms
```

The window is nowhere near any sysctl ceiling. It is small because the flow is
**app-limited**: bursty request/response that never sustains enough data in
flight for autotuning to open further.

**Important caveat against over-reading this.** `rcv_space` tracks what the
application consumes per RTT, so finding it near our ~1.4 MB request size is a
*consequence*, not a cause. It is circular to call it the limit.

And ROOT's own sockets show `rcv_space` of 1.24 / 1.33 / 2.06 MB and are
**also `app_limited`** — the same regime. So "ROOT fills the pipe better, so its
window grows" is not supported either.

Measured request concurrency, AGC run against FNAL: cache **0.98** in flight,
ROOT direct **2.45**. At UCSD, where there is no penalty, both are low
(0.41 and 0.58). That correlation is suggestive but unproven.

## Connection and concurrency, also eliminated

| leg | connections observed |
|---|---|
| ROOT direct -> FNAL | 12 (one per pool node) |
| ROOT -> cache | 2 |
| cache -> FNAL | 10 (one per pool node) |

The cache collapses 9 origin endpoints onto one host, so the *client* side drops
from 12 connections to 2 -- but the cache's own outbound side keeps 10. Raising
the client side with `XRD_SUBSTREAMSPERCHANNEL=8` gives 16 connections and
changes nothing (237.4 s vs 239.7 s). ROOT itself only ever has ~2.45 requests
in flight, so extra connections add no extra requests. Concurrency is not the
constraint on either leg.

`pss.setopt` notes, since the docs are stale: `ParStreamsPerPhyConn` is a v4
alias parsed in `XrdOucPsx::ParseSet` and forwarded to XrdCl's
`SubStreamsPerChannel` (stock default 1, pss does not override it). pss *raises*
`WorkerThreads` from XrdCl's stock 3 to **64** and sets `ParallelEvtLoop` to 10,
matching stock -- so no pss default is lower than XrdCl's own.

Worth raising with Andy independently of any of this: **`do_Read` has a full aio
path and `do_ReadV` has none.** Vector reads therefore cannot overlap on a link
and are immune to every `xrootd.async` setting. That is invisible on a LAN and
matters at WAN latency, and vector reads are the dominant access pattern for
ROOT analysis.

## Where to go next, at this point

*Also historical. The packet capture below was done, and "raise origin-request
concurrency" turned out to be the right instinct for the wrong reason: the
cache was not failing to pipeline, it was never given more than one client
vector read to work on. The note on `xrootd.async segsize` still holds.*

The client side is exhausted. Distinguishing the remaining possibilities needs
either a packet capture (in-flight bytes over time, ACK pattern, retransmits) —
which needs root — or origin-side logs. Two specific things worth testing:

1. **Raise origin-request concurrency in XrdPfc.** The cache runs at ~1 request
   in flight against a path whose BDP is ~6 MB. Deep pipelining, rather than
   one origin readv per client readv, would keep the flow from being
   app-limited. This is the most promising lead and is a cache-side change.
2. Whether a sustained, non-bursty transfer to the same origin reaches full
   BDP. If it does, app-limiting is confirmed as the regime; if it does not,
   something else caps the path.

Note on Andy's 64 kB `xrootd.async segsize`: that governs *regular* read
segmentation, a different mechanism which vector reads bypass entirely. It has
no bearing on any of the above.

## Reproducing

`test/rvbench.cxx` issues N vector reads of C chunks and reports per-request
latency; build it against both the local tree and the system XrdCl with
`test/pgrv_build.sh`-style flags. Measure the cache's *origin-side* latency from
its own XrdCl log (`XRD_LOGLEVEL=Debug XRD_LOGFILE=...`), matching
`MsgHandler created` to `Calling MsgHandler` by handler address — comparing a
client-side number against an origin-side one is the easiest mistake to make
here, and I made it.

Beware also: measurements against EOS and dCache are extremely noisy (EOS direct
swept 159 to 639 ms non-monotonically), and anything measured through an ssh
tunnel picks up OpenSSH's own channel window — a clean-looking 1.3 MB "step"
measured that way turned out to be an artifact, absent at 9 ms direct.

### Both legs, now actually tested

The two legs have independent concurrency limits, and the connection counts
that looked asymmetric are topology, not client policy.

ROOT applies the same policy to the cache and to FNAL -- it has no idea which
it is talking to. The 12-vs-2 difference is simply that FNAL redirects onto a
dCache pool: 10 of the cache's 11 outbound sockets go to distinct pool nodes,
one channel each.

```
[2620:6a:0:8420::101]:1094      dCache door
[2620:6a:0:8421::a7]            9 distinct pool nodes
[2620:6a:0:8420:f0:0:205:40]    ...
```

| leg | knob | connections | execute |
|---|---|---|---|
| ROOT -> cache | `XRD_SUBSTREAMSPERCHANNEL=1` | 2 | 239.7 s |
| ROOT -> cache | `XRD_SUBSTREAMSPERCHANNEL=8` | 16 | 237.4 s |
| cache -> FNAL | default | 10 | 237.7 s |
| cache -> FNAL | `pss.setopt ParStreamsPerPhyConn 8` | 11 | 226.8 s |

All four inside the noise band for identical configs (222.9 -- 264 s).

Two traps worth writing down:

* **`XRD_PARSTREAMSPERPHYCONN` is not a variable.** XrdCl derives env names
  mechanically as `XRD_` + registered name (`XrdClDefaultEnv.cc:415`), and the
  registered name is `SubStreamsPerChannel` (`:270`). `ParStreamsPerPhyConn`
  exists only as a v4-compat alias in the pss config table
  (`XrdOucPsx.cc:702-733`), translated before it reaches XrdCl. Setting the
  env form is a silent no-op -- it produced 264.0 / 260.7 s with the
  connection count pinned at 2 in both runs.

* **Substreams are created lazily.** 8 substreams x 10 channels should allow
  80 sockets; it produced one extra. XrdCl opens a substream only when a
  channel is genuinely backed up, and at ~2.5 requests in flight over 10 pool
  nodes none ever is. The knob has nothing to relieve.

Concurrency is eliminated on both legs.

## Partly resolved: cwnd reset after idle (real, but not the 8x)

Packet capture on `black` (`enp25s0`, `net 2620:6a::/32`), cold AGC run through
the cache vs ROOT direct, same box, same link, same clock. 229.0 s vs 29.0 s, so
the capture caught the real thing.

RTT to FNAL is **57.8 ms** (TCP handshake), not the ~9 ms of the UCSD origin.

|  | cache -> FNAL | ROOT direct -> FNAL |
|---|---|---|
| connections | 10 | 12 |
| bytes from origin | 862 MB | 775 MB |
| think time (request -> first response byte) | 56.9 ms | 56.0 ms |
| transfer (first -> last response byte) | **336.1 ms** | **55.6 ms** |
| throughput *while bytes are flowing* | 91.1 MB/s | 74.2 MB/s |
| dead air (gaps > 5 ms) | **166.5 s** | 44.5 s |
| idle between response and next request | **2151 ms** | 66 ms |
| ...fraction over 200 ms (min RTO) | **68%** | 14% |
| bytes in first round of a response | **14 kB** | 294 kB |
| RTT-sized rounds per response | **10** | 1 |

Think time is identical and equals one RTT: **dCache is not slow**, and the
door/pool is not the problem. Throughput while flowing is *higher* for the
cache. The entire difference is idle wire.

The chain:

1. The cache keeps ~2.5 requests in flight, spread over 10 pool connections.
2. Each connection therefore idles ~2.1 s between requests, past the 200 ms
   minimum RTO on 68% of occasions.
3. `net.ipv4.tcp_slow_start_after_idle = 1` resets cwnd to the initial window
   every time that happens.
4. The measured first round is 14 kB. IPv6, MTU 1500, TCP timestamps -> MSS
   1428; IW10 = 14280 B. It is the initial window exactly.
5. A ~1 MB response then needs ~10 RTT rounds to ramp. 336 ms of ramp + 56.9 ms
   think = 393 ms, which is the 415 ms readv latency measured from the logs.

So it is a window effect -- but the **congestion** window, not the receive
window, and the trigger is idleness, not buffer size. The earlier `rcv_space`
reasoning was circular and wrong.

This also explains why every concurrency knob failed. More connections and more
substreams spread the same few requests thinner, lengthening per-connection
idleness. We were pushing the wrong way the whole time.

### Consequences for XCache

* `net.ipv4.tcp_slow_start_after_idle=0` on cache hosts is a free, one-line win
  on any high-RTT origin. Worth putting in the deployment notes.
* Fewer, busier connections beat more, idler ones -- the opposite of the usual
  instinct. Anything that keeps a pool connection warm is worth more than extra
  parallelism.
* This is an argument *for* larger `pfc.iosize` and for pgReadV batching beyond
  the obvious request-count saving: bigger requests amortise the ramp, and a
  steady request stream keeps cwnd from collapsing at all.
* It is a general WAN-cache result, not specific to dCache or to FNAL.

Capture and analysis scripts: `test/wan-analysis/` (see its README).
Instrumentation: `NETTIME` trace lines at `pfc.trace info`, `File::TraceNetTime()`
in `XrdPfcFile.cc`, CLOCK_REALTIME so they join against frame timestamps.

## Correction: the network was not the answer

`tcp_slow_start_after_idle` was set on `black`, which is the **wrong side**. It
governs the local *sender's* cwnd; the bulk data flows FNAL -> us, so the window
that resets is dCache's, on their kernel. Our request packets are the only thing
our setting touches. The first round stayed at exactly 14 kB after the change,
which is how we know.

Raising client concurrency (`--ncores` 8 -> 24) *did* keep the pool connections
warm, and it fixed the ramp completely:

| | baseline | ssai=0 | ncores 24 | ROOT direct |
|---|---|---|---|---|
| think | 56.9 ms | 56.9 ms | 57.3 ms | 56.2 ms |
| transfer | 336.1 ms | 222.0 ms | **60.8 ms** | 58.9 ms |
| first round | 14 kB | 14 kB | **71 kB** | 289 kB |
| rounds/response | 10 | 5 | **2** | 2 |
| conn idle | 2151 ms | 1225 ms | **479 ms** | 59 ms |
| total | 229.0 s | 202.9 s | **245.0 s** | 28.5 s |

Per response the cache now matches ROOT direct exactly. The run got *slower*.

### What actually limits it

From the NETTIME logs, across all three runs:

```
MEAN requests in flight: 1.00      peak 8
```

Exactly 1.00, with 787 / 787 / 1347 requests at medians of 393 / 286 / 168 ms.
The run is `n_requests x latency / 1`, so halving latency bought nothing.

It is not a lock in our code (`m_state_cond` is released at `XrdPfcFile.cc:1279`,
before the request is issued at `:1284`), not RAM (`pfc.ram 4g` against 862 MB
fetched, no starvation in the log), and not one-file-at-a-time: 6-7 files are
being fetched concurrently, peak 9.

The arithmetic closes instead on the per-file cycle. Each file issues a remote
request every **2.2 s**, of which **0.393 s** is the fetch. 7 files x
(0.393 / 2.2) = 1.0 in flight. The files are independent and each is slow.
**82% of the cycle is not spent waiting on the network.**

Consistent with the warm cache running this same analysis in 11.7 s -- faster
than direct's 28.4 s. The steady-state read path is fine. The cost lives in the
cold path, alongside 210,358 four-kilobyte block writes.

### Next step

Extend the NETTIME approach to the cold path: stamp response arrival ->
`WriteRunToDisk` -> block release -> `FinalizeReadRequest`, and find where the
1.8 s per file per cycle goes. That is cache-internal and entirely ours.

## Resolved: concurrency equals the number of client sessions (mechanism below)

Instrumented every client read (`CLIREAD`) as well as every remote request
(`NETTIME`). In the cold run there were **787 client vector reads and 787 origin
requests, 1:1**, and the cache logged **one login**.

```
MEAN client reads in flight : 0.99   peak 8
MEAN remote reqs in flight  : 0.99   peak 8
```

Splitting the identical cold workload across three independent client processes,
measured over the window where all three overlap:

```
 t(s)  files   mean in flight
    0      6            3.22
   10      7            3.19
   15      6            2.97
   20      6            1.83   <- two processes finish
   25      5            0.99   <- one session left
   ...     5            0.99
```

Mean in-flight tracks the **number of client processes, exactly**. Open files
(5-7) and RDataFrame threads (8) make no difference at all.

The arithmetic then closes on every measurement taken today:

| | client reads | per read | predicted | observed |
|---|---|---|---|---|
| warm cache | 787 | 14.1 ms | 11.8 s | **11.1 s** |
| cold cache | 787 | 282.8 ms | 222 s | **178-245 s** |
| ROOT direct | 787 | ~56 ms over 12 pool sessions | ~28 s | **28.5 s** |

Direct to dCache, ROOT holds a session per pool node and overlaps across them.
Through the cache -- a single endpoint -- it holds one. That is the whole 8x.

**Not attributed to `do_ReadV`.** It is tempting, since `do_ReadV`
(`XrdXrootdXeq.cc:2746`) has no aio path where `do_Read` has the full
`xrootd.async` machinery, and `xrootd.async off`/`nocache` measurably change
nothing (241.1 / 240.7 s). But with a single session the *peak* was 8, so the
server did handle eight concurrent vector reads on one link. Whatever pins the
mean at 1.0 is upstream -- ROOT's I/O path or XrdCl per-session handling -- and
is not yet isolated.

### Not confined to caches

The measured condition is *one client, one server endpoint, several concurrent
vector reads*. A multi-threaded RDF/RNTuple job over one large file meets it
without any cache: per-thread `TFile`s still pool onto a single channel, since
the key is `(user, host, port)`. Directly measured — eight `XrdCl::File` handles
on the same path, one process, two sockets, 5.13 s; separate channels, sixteen
sockets, 3.82 s. Straight to one xrootd server or EOS, likewise. Behind a
redirector it depends on whether opens spread across data servers.

### What this means for XCache

Interposing a single-endpoint cache in front of a distributed origin collapses
the parallelism the origin's architecture was providing. Warm this is invisible
(14 ms x 787 = 11 s, still beating direct). Cold over a WAN it is decisive.
Directions worth exploring, roughly in order of expected value:

1. Find and remove the per-session serialization (needs isolating ROOT vs XrdCl
   first -- it may not be ours at all).
2. Prefetch/readahead, so a client read finds data already in RAM and the RTT is
   never on the critical path. `pfc.prefetch` is off for this use case today.
3. Multiple cache endpoints, so a client can hold several sessions.

Fewer round trips also matters more than we thought, since each one is fully
exposed -- which is the strongest available argument for pgReadV batching.

## SOLVED: one vector read in flight per XrdCl channel

`do_ReadV()` runs synchronously on the link's thread, so a link carries exactly
one vector read at a time. XrdCl pools channels on `(user, host, port)`, so a
client reading N files from one cache gets **one** channel and therefore no read
concurrency at all. Going direct to a distributed origin, the N files land on N
different pool hosts, so the same client gets N channels and N-way concurrency.
That asymmetry is the entire 8x.

### Proof, in three steps

**1. It is not ROOT.** A bare XrdCl probe (`test/rvconc.cxx`) asking for K
concurrent vector reads on one session:

```
conc=1  30 reqs  wall= 2.34 s   78.0 ms/req
conc=4  30 reqs  wall= 3.46 s  115.2 ms/req
conc=8  30 reqs  wall= 2.79 s   92.9 ms/req
```

K=8 should have been ~0.3 s. No overlap. No ROOT involved.

**2. It is not pss / posix / pfc.** `CLIREAD` counts client-facing reads inside
the cache, above all of them:

```
MEAN client reads in flight : 0.98   peak 1
MEAN remote reqs in flight  : 0.97   peak 1
```

Peak **1**. The cache never saw two client vector reads at once while the client
had eight outstanding, so the serialization is above the cache.

**3. It is not XrdCl either.** Loopback capture, K=8:

```
0.0077 REQ->    1048     eight readv requests pipelined onto one
0.0077 REQ->    1048     socket within 100 us, before any response
   ...  (x8)
0.8678 <-RSP   32768     860 ms later, responses arrive together
```

XrdCl sends all eight immediately. The server takes 860 ms to do eight, where
one alone takes 78 ms. The server received the concurrency and did not use it.

### The workaround: distinct usernames, no code change

XrdCl's channel key includes the username, so pooling can be defeated from
outside by giving each file URL a different user. Full AGC cold run:

| | logins | sockets | in-flight | execute |
|---|---|---|---|---|
| `root://localhost:7987/...` | 1 | 2 | 1.00 | **249.6 s** |
| `root://u<i>@localhost:7987/...` | 9 | 18 | 2.64 | **36.7 / 36.5 / 26.4 s** |

Three cold repeats: 36.7 / 36.5 / 26.4 s. ROOT direct is 28.5 s, so the cold
cache goes from **8.8x behind to parity** -- one repeat beat direct outright.
Nothing patched, no lock touched, no XrdCl change, and the physics is identical
(124 keys, 52039 entries, integral 93931.1219 in both).

Microbenchmark agrees: 8 file handles in one process, 6.47 s shared channel vs
3.27 s with distinct usernames; eight separate *processes* give 2.43 s.

### Thread knobs are irrelevant

`ParallelEvtLoop` (`XrdClPollerBuiltIn.cc:708`) is the **poller** count, default
**10**. `WorkerThreads` is the **worker pool**, default **3**; XrdPss sets it to
64 for the cache's own client and leaves the pollers at 10. Neither moves this
workload -- with one outstanding request per channel there is nothing to
parallelise. (Note: a test here set `ParallelEvtLoop=8`, which *lowers* it from
the default 10. It made things marginally worse, as expected.)

### What to do about it

1. **Now, for anyone wanting to see real XCache performance:** distinct
   usernames per file, or several cache endpoints. Purely client-side.
2. **Properly:** give `do_ReadV` an async path like `do_Read` has. This is the
   invasive one and needs Andy.
3. Prefetch would also hide it, by taking the RTT off the critical path.

Probes: `test/rvconc.cxx` (K concurrent readv, one session), `test/rvmulti.cxx`
(N file handles, optional distinct usernames).

## The mechanism, exactly: `readv.pathid` is ignored

`ClientReadVRequest` in `XProtocol.hh` **already carries a pathid byte**:

```c
struct ClientReadVRequest {
   kXR_char  streamid[2];
   kXR_unt16 requestid;
   kXR_char  reserved[15];
   kXR_char  pathid;      // <-- on the wire since forever
   kXR_int32 dlen;
};
```

Nothing in the tree references `readv.pathid` -- not the server, not the client.
It is a declared, unused field. Meanwhile every other bulk request honours its
pathID and offloads:

| request | offload |
|---|---|
| `do_Read` | `XrdXrootdXeq.cc:2624` -> `do_Offload(&do_ReadAll, pathID)` + aio |
| `do_PgRead` | `XrdXrootdXeqPgrw.cc:199` -> `do_Offload(&do_PgRIO, pathID)` |
| `do_Write` | `:3352`, `:3361` -> offload |
| **`do_ReadV`** | **none; never parses pathID** |

`do_PgRead` takes its pathID from `ClientPgReadReqArgs`, so **a `do_PgReadV`
modelled on it inherits offload for free.**

This retroactively explains the morning's dead end: `XRD_SUBSTREAMSPERCHANNEL=8`
gave 16 connections and changed nothing. Substreams *are* parallel paths --
pathIDs -- and read/pgread/write all use them. readv discards the field, so every
vector read stays on the main link. The knob worked; the request type could not
use it.

**The fix needs no protocol change.** The byte exists; `do_ReadV` just has to
honour it the way `do_Read` does.

## Workarounds, ranked

XrdCl keys channels in `URL::GetChannelId()` on `protocol://hostId/` plus a few
CGI keys -- one of which is `xrdcl.intent`. Full AGC cold run against FNAL:

| | logins | in-flight | execute |
|---|---|---|---|
| default | 1 | 1.00 | **249.6 / 205.6 s** |
| distinct username per file | 9 | 2.64 | **36.7 / 36.5 / 26.4 s** |
| `?xrdcl.intent=chanN` | 10 | 1.85 | **47.8 s** |
| ROOT direct | -- | -- | 28.5 s |

Physics is bit-identical in all cases (124 keys, 52039 entries, integral
93931.1219), and the CGI does **not** leak into cache file names, so cache keys
and warm hits are unaffected.

**Prefer `xrdcl.intent`** over usernames: no fake identities, so no per-user
authentication, mapping, quota or monitoring side effects. Under GSI or tokens
N usernames would mean N authentications.

### An env var was tried and is the wrong answer

Added `ChannelsPerHost` to XrdCl (constant, env registration, a per-URL-instance
slot appended in `GetChannelId`). It works in a native client: sockets scale
2/4/8/16 with N, and N=8 gives 3.82 s against 5.13 s pooled, matching the URL
tricks. It is **useless in practice**, because ROOT's `libNetxNG.so` binds
`libXrdCl.so.3` while this tree builds soname 5. Only a client rebuilt against
new XrdCl would see it.

The URL tricks are pure URL *content*, so they work with any XrdCl version --
which is why the 249.6 -> 26 s result was obtainable at all.

**Trap worth remembering:** `build/lib` contained a stale soname generation from
2026-06-09 -- `libXrdCl.so.3` and friends -- alongside today's soname 5. ROOT
links the *soname* `libXrdCl.so.3`, so `LD_LIBRARY_PATH=build/lib` made the
loader prefer the June copy over `/usr/lib64`, and ROOT silently ran a
three-month-old client for every client-side measurement here. Cleared on
2026-09-02 into `build/lib-stale-pre-soname5/`.

ROOT still loads two generations at once -- soname 3 from `/usr/lib64` for
`libNetxNG`, soname 5 from our build via `libXrdClHttp-5.so`. So any client-side
XrdCl change must be checked with `strings` or `/proc/<pid>/maps`, never
assumed.

### The residual limit

A single file opened once cannot be helped by anything client-side: one link,
one vector read at a time, by construction. Multiple *opens* of one file can be
split (measured: 8 handles on one path, 4.97 s pooled vs 3.91 s split), but a
one-file analysis has nothing to spread. That case needs the server fix, which
is the real argument for honouring `readv.pathid`.

## Why readv was left synchronous, and where the decision belongs

Not an oversight. Throwing N concurrent vector reads at a classic xrootd data
server is bad, especially several on the same file when that file lives on one
spinning disk -- the resulting seek pattern is exactly what a vectored read
exists to avoid. Leaving `do_ReadV` synchronous is a defensible default for a
plain data server.

But it is a property of the **storage**, not of the request, and a cache cannot
answer it statically either. One XCache serves:

* a cold file from a WAN origin -- concurrency is free and worth ~8x;
* a fully-cached file from a local spinning disk -- concurrency is harmful;

sometimes inside the same request. So neither the protocol layer nor a static
server config can decide. The decision has to be made where the storage is
known: in the OSS / cache, at request time.

That reframes the ask to Andy from "make readv async" to "make the async path
reachable and delegate the choice downward". Written up in
`xcache-step2-analysis-environment.md`.

### Why a server-side knob is worth more than a client-side one

ROOT and XrdCl reach users through central builds (CERN, LCG, CMS) that a user
cannot change -- they run whatever their environment ships. An XCache is always
deployed by the site, so a toggle there can actually be flipped. This is why the
`ChannelsPerHost` XrdCl prototype, even though it works, is the wrong answer:
it only helps clients rebuilt against new XrdCl, and ROOT 6.38 binds
`libXrdCl.so.3`.

The URL tricks (`xrdcl.intent`, usernames) are the exception that works today,
because they are pure URL *content* and need no new client code at all.

### Mechanism notes for a future attempt

* `do_Read` has **two** async routes: aio on the same link when `pathID==0`
  (`XrdXrootdXeq.cc:2597-2616`, ending in `return 0`), and `do_Offload` when a
  pathID is supplied (`:2624`). The first needs no client cooperation at all --
  that is the one a server-side toggle would use.
* Throttling already exists: `as_maxperlnk`, `as_maxpersrv`, from
  `xrootd.async limit` / `maxtot` (`XrdXrootdConfig.cc:983-987`).
* `XrdXrootdAioFob` is already per-file -- the natural home for "one outstanding
  vectored read per LFN", which is the granularity that recovered 7-9x via the
  channel workaround while keeping a single file's readvs serialised.
* `XrdXrootdAioTask::Read(long long offs, int dlen)` is single-extent. A vector
  form needs a new `XrdXrootdAioTask` subclass or per-element completion
  aggregation. This is the real work and presumably why it never happened.
* `do_PgRead` already offloads, so a `do_PgReadV` modelled on it inherits the
  right behaviour. The gap is specific to plain `kXR_readv`.

## Code

Diffs against upstream `master`, on `github.com/osschar/xrootd`:

* [`pfc-optimize-for-small-blocks`](https://github.com/xrootd/xrootd/compare/master...osschar:pfc-optimize-for-small-blocks) — block runs and the
  NETTIME/CLIREAD instrumentation (`kTraceNetTime`, off by default) that every
  in-flight measurement here is derived from.
* [`xrdcl-channels-per-host`](https://github.com/xrootd/xrootd/compare/master...osschar:xrdcl-channels-per-host) — the client-side channel-splitting
  prototype.

Probes and analysis scripts are in the working tree, not on those branches:
`test/rvconc.cxx`, `test/rvmulti.cxx`, `test/wan-analysis/`.

## Status, 2026-09-03

Andy is generally supportive; he will review the branches. Three things came out
of the first conversation:

* **`kXR_pgreadv` may not be needed.** The per-element `fhandle` in
  `readahead_list` is dead weight in the readv *response*, and crc32c is exactly
  4 bytes. Under a flag -- there are 15 unreferenced `reserved` bytes in
  `ClientReadVRequest` -- readv can carry a checksum per iochunk instead. See
  "Andy's alternative" in Step 1.
* **A server can split channels by itself**, by redirecting an open to a
  synthetic username. `SFS_REDIRECT` -> `kXR_redirect` -> `URL::ParseHostInfo`
  -> `ComputeHostId` -> channel key. **Measured 2026-09-03: 276 s -> 31 s with
  an unmodified client fileset, mean in flight 1.00 -> 2.83, i.e. parity with
  not caching at all.** No client change of any kind. See Step 2.
* **Async readv stays open**, pending review of the existing branches.

Follow-up 2026-09-03: raising the redirect inside the cache (pss layer, early
open) needs no protocol change -- `-EDESTADDRREQ` plus `FileURL` in the open
env is already wired through `XrdOfs.cc:788`. But a self-redirect to the same
`host:port` differing only in username loops on a stock client, because
`RetryAtServer` compares `URL::GetLocation()`, which omits the user. Comparing
`GetChannelId()` too is a one-line fix. Redirecting across loopback addresses
avoids the client change and does create the channels, but did not produce the
concurrency. Since the same usernames placed directly in the client's fileset
URLs against that same single instance do work, this is a mechanism in XrdCl's
redirect processing rather than anything topological. The 276 s -> 31 s result
used a genuinely different endpoint.