# Step 2 — Making it work in an analysis environment

*Concurrent `kXR_readv`.*

**For:** Andy Hanushevsky
**From:** Matevž Tadel (with analysis by Claude, working in the xrootd tree)
**Date:** 2026-09-02
**Depends on:** Step 1 (`xcache-step1-small-reads.md`), which stands as written.

Step 1 makes small-block caching efficient. It does not make it fast enough to
use. This document covers what is needed for that, and adds a second request to
the one Step 1 made.

---

## What Step 1 asked for, in one line

`kXR_pgreadv`: N `(offset, length)` pairs like `kXR_readv`, crc32c per 4 kB page
like `kXR_pgread`, because `pfc.cschk net` currently forces XCache into one
origin request per cache block and there is no vector form to batch them into.

That ask is unchanged, and Step 1's implementation notes still hold — it is built and
working on branch `pfc-pgreadv`.

## Measurements over a wide-area path

Step 1's numbers came from a UCSD origin at 8 ms RTT, where per-request cost is small
enough that request *count* looks like an efficiency question. We have since run
the IRIS-HEP AGC ttbar analysis (ROOT RDataFrame, TTreeCache, CMS open data
NanoAODv9) against **FNAL dCache at 57.8 ms RTT**, with `tcpdump` on the cache's
outbound leg and nanosecond instrumentation inside XrdPfc.

Identical physics, identical client, same host and link:

| | client-visible | |
|---|---|---|
| ROOT direct to FNAL | **28.5 s** | 12 connections, to 12 dCache pool nodes |
| through XCache, warm | **11.1 s** | faster than direct — the cache is good at its job |
| through XCache, cold | **178–245 s** | 6–8x slower than direct |

The cold number is the problem, and it is *not* what we assumed.

### The network is not the bottleneck

From the packet capture, per origin response:

| | XCache -> FNAL | ROOT direct -> FNAL |
|---|---|---|
| think time (request -> first response byte) | 56.9 ms | 56.0 ms |
| throughput while bytes are actually flowing | 91.1 MB/s | 74.2 MB/s |
| dead air (inter-arrival gaps > 5 ms) | 166.5 s | 44.5 s |

Think time is one RTT and is *identical*. dCache is not slow. The wire is
slightly faster for the cache than for ROOT. The entire difference was idle time.

We found a real cause for part of it — the cache's pool connections idle ~2.1 s
between requests, past the minimum RTO, so the origin's congestion window resets
to IW10 (measured first round: 14 kB, exactly 10 x 1428 MSS) and each ~1 MB
response spends ~10 RTTs ramping. Raising client concurrency warmed the
connections and fixed it completely: transfer time per response fell from 336 ms
to **60.8 ms, versus ROOT direct's 58.9 ms — parity**.

**The run got slower.** 245 s instead of 229 s.

### The actual bottleneck: no request overlap through a cache

The benchmark runs **8 RDataFrame threads over 9 input files**, and the cache
confirms it has them all: 9 distinct `IO` objects for 9 distinct files, 6-7 of
them being fetched concurrently, peak 9. There is no shortage of independent
work to overlap.

Instrumenting every remote request with wall-clock issue/completion timestamps:

```
MEAN remote requests in flight : 1.00      peak 8
MEAN client reads in flight    : 0.99      peak 8
```

Exactly 1.00, across three configurations with 787 / 787 / 1347 requests at
median latencies of 393 / 286 / 168 ms. The run is `n_requests x latency / 1`,
which is why halving latency bought nothing at all.

The client-side counter is the telling one. In the cold run there were **787
client vector reads and 787 origin requests — 1:1**, and the cache logged
**one login**. ROOT opens a single session to the cache, all nine files ride it,
and vector reads on it do not overlap. Going direct, the same ROOT process gets
twelve links to twelve pool nodes and overlaps freely.

The arithmetic then closes on every measurement we have:

| | client reads | per read | predicted | observed |
|---|---|---|---|---|
| warm cache | 787 | 14.1 ms | 11.8 s | **11.1 s** |
| cold cache | 787 | 282.8 ms | 222 s | **178–245 s** |
| ROOT direct | 787 | ~56 ms over 12 links | ~28 s | **28.5 s** |

The rule, stated plainly and confirmed by the split-process test below:
**a client's outstanding-request concurrency equals the number of server sessions
it holds, and nothing else.** Going direct to dCache, ROOT gets a session per pool
node — twelve of them — and overlaps across them. An XCache is a *single
endpoint*, so the same client gets one session and one outstanding request.

This is not a tuning problem, it is structural: interposing any single-endpoint
cache in front of a distributed origin collapses exactly the parallelism the
origin's own architecture was providing. Warm, this is invisible (14 ms x 787 = 11 s,
still beating direct). Cold, over a WAN, it is the whole story.

## Effect on the pgReadV request

If round trips are serialized, their **count** is the only thing that matters,
and that is precisely what pgReadV controls. On this workload, at
`pfc.blocksize 4k` and `pfc.iosize 1m`:

| | origin round trips |
|---|---|
| `pfc.cschk off` — one `kXR_readv` per client vector read | **587** |
| `pfc.cschk net` — forced to one `kXR_pgread` per run | **13,405** |

23x more round trips. At the measured 282 ms per serialized round trip that is
not "somewhat slower", it is roughly an hour instead of three minutes. Without
`kXR_pgreadv`, **end-to-end checksums are not merely expensive on a WAN cache at
small block size — they are unusable.** Step 1 argued this on efficiency grounds; the
WAN data turns it into a feasibility argument.

## The second ask: make the async path reachable for `do_ReadV`

The cause is in the server. Making vector reads asynchronous unconditionally
would be wrong, for the reason that presumably kept them synchronous. The
request is narrower: make the async path **reachable**, and let the layer that
knows the storage decide whether to take it.

### What we found

`ClientReadVRequest` in `XProtocol.hh` already carries a `pathid` byte:

```c
struct ClientReadVRequest {
   kXR_char  streamid[2];
   kXR_unt16 requestid;
   kXR_char  reserved[15];
   kXR_char  pathid;      // on the wire, referenced nowhere in the tree
   kXR_int32 dlen;
};
```

Nothing in the tree reads it -- not server, not client. Meanwhile:

| request | async mechanisms |
|---|---|
| `do_Read` | aio on the same link (`pathID==0`, `XrdXrootdXeq.cc:2597-2616`, `return 0`) **and** `do_Offload` for `pathID!=0` (`:2624`) |
| `do_PgRead` | `do_Offload(&do_PgRIO, pathID)` (`XrdXrootdXeqPgrw.cc:199`), pathID from `ClientPgReadReqArgs` |
| `do_Write` | `do_Offload` (`:3352`, `:3361`) |
| **`do_ReadV`** | **neither** |

`do_ReadV` therefore completes synchronously on the link's thread, so one link
carries one vector read at a time. Three measurements pin it there:

* **Not ROOT.** A bare XrdCl probe issuing K concurrent vector reads on one
  session: K=1 gives 78 ms/req over 30 requests; K=8 gives 93 ms/req. No overlap.
* **Not pss / posix / pfc.** Instrumenting the cache's client-facing reads, which
  sit above all three: `MEAN 0.98, peak 1`. The cache never saw two client vector
  reads at once while eight were outstanding.
* **Not XrdCl.** Loopback capture, K=8: eight readv requests pipelined onto one
  socket inside 100 us, before any response; first response 860 ms later. XrdCl
  delivered the concurrency; the server did not use it.

This also explains why `SubStreamsPerChannel` does nothing for a readv workload
(we measured 2 -> 16 connections, no change). Substreams *are* parallel paths;
read, pgread and write all use them, and readv discards the field.

**Note for pgReadV:** because `do_PgRead` already honours pathID and offloads, a
`do_PgReadV` modelled on it inherits the right behaviour for free. The problem is
specific to plain `kXR_readv`.

### It is not only a cache problem

An XCache is a guaranteed instance of this, not the only one. The condition is
just *one client, one server endpoint, several concurrent vector reads*, and
that is easy to meet with no cache in sight:

* **A multi-threaded RDF or RNTuple job over a single large file.** Threads work
  on different parts of the file, and each may open its own `TFile` — but XrdCl
  keys channels on `(user, host, port)`, so every one of those opens lands on
  the same channel. Measured directly: eight `XrdCl::File` handles on the *same
  path* from one process share two sockets and gain nothing from concurrency
  (5.13 s); forcing them onto separate channels gives sixteen sockets and
  3.82 s.
* **A job talking straight to one xrootd server**, EOS included.
* **Through a redirector it depends.** If the opens are spread across data
  servers, the client gets a channel per server and the limit stays hidden. If
  they all land on the same server — a single replica, or a redirector that
  pins them — it appears exactly as it does in front of a cache.

The `?xrdcl.intent=` workaround applies unchanged in every one of these: it is a
property of the URL, not of the cache.

### A server-side way to split channels

Andy pointed out that a server can hand a connecting client a different
username, which is enough to make XrdCl open a new socket. Traced through the
code, it works, and it is better than anything client-side:

1. An OSS/OFS plugin returns `SFS_REDIRECT` with the ErrInfo code as the port
   and the message as the host (`XrdSfsInterface.hh:115`).
2. That reaches the client as `kXR_redirect`, message passed through
   (`XrdXrootdXeq.cc:3936`).
3. XrdCl builds a `URL` from it, and `URL::ParseHostInfo` accepts
   `user[:pass]@host`.
4. `URL::ComputeHostId()` puts `user@` into `pHostId`, and `GetChannelId()` is
   `protocol://pHostId/` -- so the redirect lands on a **new channel**.

So a cache can redirect each open to itself under a synthetic username and get
one channel per open, with no client change, no protocol change, and nothing
for the user to do. That is strictly better than asking analysers to rewrite
their filesets with `?xrdcl.intent=`.

Note that XrdCl inherits the client's original username *only* when the
redirect omits one, so a redirect-supplied name wins.

#### Measured

Tested on 2026-09-03 with a bare redirector on one port bouncing read opens to
the cache on another, and the client fileset left **completely unmodified**.
The static `xrootd.redirect` directive cannot express a username -- `xred_php()`
validates the target with `XrdNetAddr::Set()`, so `u1@localhost` is rejected --
so the rotation was done with a throwaway patch at the redirect send.

| | logins | mean in flight | execute |
|---|---|---|---|
| stock redirect, client's own username | 1 | **1.00** | 275.9 / 309.1 s |
| redirect rotating 8 synthetic usernames | 8 | **2.83** | **31.4 / 29.5 s** |

787 remote requests either way, so the work is identical. ROOT direct to FNAL
is 28.5 s: **the redirect brings a cold cache to parity with not caching at
all**, and asks nothing of the client. That is a ~9x improvement from a server
change alone, and it beats both URL-level workarounds (`xrdcl.intent` gave
47.8 s, a distinct username per file 26-37 s).

In flight settles at 2.83 rather than 8 because per-response latency falls once
requests overlap -- the connections stay warm, so the congestion window does
not collapse between requests. 787 requests at ~113 ms over 2.83 concurrent is
the 31 s.

#### Where it has to be raised, and one thing it runs into

The plumbing to raise this inside the cache -- rather than from a separate
redirector -- already exists, and needs no protocol change:
`XrdPssFile::Open` puts a full URL into the open env and returns
`-EDESTADDRREQ`; `XrdOfs.cc:788` turns that into `SFS_REDIRECT` with
`ecode = -1`; the protocol layer sends the URL as the redirect body; XrdCl
parses `user@host` out of it, and `URL::GetChannelId()` includes the user.
`XrdPssFile::Open` already has `Env.secEnv()` in hand, so the loop guard is a
comparison -- though it has to look at `tident`, not `name`: under
`sec.protocol host` there is no authenticated name.

**But a self-redirect to the same `host:port` differing only in username loops
on a stock client.** `XRootDMsgHandler::RetryAtServer` gates its "new server"
branch on `URL::GetLocation()`, which is `protocol://host:port/path` and
deliberately omits the user. The redirect therefore looks like the same
server, the request is retried on the existing channel, the server redirects
again, and the client hits its redirect limit. Comparing `GetChannelId()` as
well is a one-line fix, and with a patched client the open succeeds and the
channels appear.

Avoiding that client change is possible: redirect to a *second port* of the
same instance (`xrd.protocol xroot:<p> +port` lets one process listen on two
ports, so `GetLocation()` differs and a stock client takes the new-server
branch). **Measured with a native XrdCl client, this works well.** Six distinct
paths, 36 vector reads, concurrency 6:

| | logins | conns to target | wall |
|---|---|---|---|
| no redirect | 1 | 0 | 15.18 s (1.9 MB/s) |
| redirect, 8-user pool | 7 (`xc0`..`xc5`) | 8 | **1.79 s (15.8 MB/s)** |

8.5x, entirely server-side. So the mechanism is sound. Two limits found:

* **Repeated opens of the same path get one channel.** Eight opens of one path
  through the redirect produced a single synthetic user: XrdCl resolves the
  redirect once and reuses that endpoint for the rest. So this does not help
  the single-large-file case either -- see below.
* **With ROOT, the redirect target must be a different process.** Full matrix,
  all runs complete, cold AGC against FNAL:

  | client | redirect raised in | topology | result |
  |---|---|---|---|
  | ROOT | protocol layer | two processes | **31 s** |
  | ROOT | pss layer | two processes | **34 s** |
  | ROOT | pss layer | one instance, 2nd port | 230 s (vs 215 baseline) |
  | ROOT | pss layer | one instance, loopback alias | 216 s (vs 254 baseline) |
  | native XrdCl | pss layer | one instance, 2nd port | 15.18 s -> 1.79 s |

  So the pss-layer hook is the right place and the mechanism is sound: two
  processes work with either mechanism. What fails is ROOT against a
  *self*-redirecting single instance. The client cannot be the cause on its own,
  since it sees byte-identical URLs in both topologies, and a native XrdCl
  client does benefit even in the single-instance case. That points at the one
  process serving both the initial and the redirected links, and it is the
  remaining question.

  Not ROOT's redirect handling, incidentally: `TNetXNGFile::Open` only inspects
  `errRedirect` on *failure* (storing `fNewUrl` for TFile to retry), so a
  redirect that XrdCl follows successfully is invisible to ROOT.

So the measured 276 s -> 31 s above stands, but it was obtained by redirecting
to a genuinely different endpoint. Whether the same-endpoint form can be made
to work without a client change is open.

Still worth measuring before production: the extra round trip per open;
authentication running again per channel, free under `sec.protocol host` but
not under GSI or tokens; and monitoring, mapping and quota all seeing synthetic
usernames. A real implementation belongs in the OSS/OFS layer returning
`SFS_REDIRECT`, where it can vary per open and be switched off, rather than in
the protocol layer's static route table.

### XrdSsi already does this, client-side

Andy pointed at XrdSsi, and it is the same trick, done in the right place.
`XrdSsiServReal::GenURL` builds the endpoint URL with a **numeric synthetic
username**:

```c
if (uEnt == 0) xUsr = xAt = "";
   else {snprintf(uBuff, sizeof(uBuff), "%d", uEnt);
         xUsr = uBuff; xAt = "@";
        }
```

giving `root://4@host:port/path`, purely so that sessions spread across
channels. `XrdSsiScale` manages the spread and answers the "how many?"
question that a fixed setting cannot: `defSprd = 4`, `maxSprd = 1024`, with
auto-tuning driven by per-channel pending counts (`minTune`, `midTune`,
`maxTune`, quadratic then linear growth).

The important part is *where*: SSI decorates the URL **client-side, at open**.
That is why it works, and it is what our fileset rewrite was doing by hand.
Measured in SSI's exact form against a single cache on a single port, with an
unmodified ROOT:

| fileset | logins | mean in flight | execute |
|---|---|---|---|
| plain URLs | 1 | 1.00 | 284.4 s |
| `root://<n>@host:port/...` | 6 | 1.37 | **76.1 s** |

No client change of any kind: the server log shows zero redirects, and ROOT
loads `/usr/lib64/libXrdCl.so.3`, so none of our XrdCl work is even in the
process. Whether our `RetryAtServer` change is present is therefore irrelevant
on this path -- it is only reached on a redirect, and there is none.

Only six distinct users appeared because the index assignment collided across
JSON entries, which incidentally shows the gain scales with the number of
*distinct* channels: nine users gave 26-37 s. That is precisely what
`XrdSsiScale` exists to tune, and an argument for adopting its logic rather
than a fixed setting.

Our redirect experiments were trying to achieve server-side what SSI achieves
client-side, which is the wrong end.

So the shape worth building is a small URL decorator, either in the
RDF/RNTuple data source or inside XrdCl so that nothing above it needs to
care -- with `XrdSsiScale` as the model for the tuning rather than a fixed N.

### The login username is not visible to the storage layer

Worth recording because it constrains any server-side variant. The login name
from `kXR_login` is sanitised and then handed only to
`Link->setID(uname, pid)` (`XrdXrootdXeq.cc:1100`). Nothing assigns it to
`XrdSecEntity::name`, which is set by authentication protocols and is null
under `sec.protocol host`. `XrdSecEntity::tident` is documented as
`//!< Trace identifier always preset` -- it merely happens to *begin* with the
login name, because that is what the link ID is built from.

So an OSS/pss-level hook cannot read the client's login name through any
documented field. Our first loop guard read `tident` and worked by accident;
a real one has to key on something explicit, such as a CGI marker placed in
the redirect URL, which does reach the storage layer -- `XrdOfs` builds the
open env as `XrdOucEnv Open_Env(info, 0, client)` from the client's opaque
info.

### Scope: this is the few-client case

An XCache in production serves O(1k) clients, each with its own session and its
own outstanding vector read. In aggregate the server is not concurrency-starved,
so the serialisation costs little there. It is a known limitation; what is new
here is the size of it in the opposite regime.

It bites when a small number of clients each want high throughput from a cold
cache. That is the interactive analysis case, and it is exactly the case small
block sizes exist for. The quantity that matters here is per-client latency, not
server aggregate throughput.

### Why it hurts caches

XrdCl keys channels on `(user, host, port)`. A client reading N files from one
XCache gets **one** channel, hence one outstanding vector read. The same client
going direct to dCache has its N files spread over N pool hosts, so it gets N
channels and N-way concurrency. Interposing a single-endpoint cache removes
exactly the parallelism the distributed origin was supplying.

Full AGC cold run, FNAL dCache at 57.8 ms RTT:

| | logins | in-flight | execute |
|---|---|---|---|
| normal URLs | 1 | 1.00 | **249.6 / 205.6 s** |
| distinct username per file | 9 | 2.64 | **36.7 / 36.5 / 26.4 s** |
| `?xrdcl.intent=chanN` | 10 | 1.85 | **47.8 s** |
| ROOT direct | -- | -- | 28.5 s |

Physics bit-identical throughout. So a **7-9x** recovery from nothing but
defeating channel pooling -- the cold cache reaches parity with direct.

### Why we think the decision belongs below the protocol

We assume readv was left synchronous deliberately. Throwing N concurrent vector
reads at a classic xrootd data server is a bad idea, especially several on the
same file, when that file lives on one spinning disk -- the seek pattern is
exactly what a vectored read is supposed to avoid. That reasoning is sound and we
are not asking you to discard it.

But it is a property of the *storage*, not of the request. For a cache in front
of remote multi-point storage it does not apply at all. And the cache cannot
answer it once and for all either: the same XCache serves a cold file from a WAN
origin (concurrency is free and worth ~8x) and a fully-cached file from a local
spinning disk (concurrency is harmful) -- sometimes within one request. So the
protocol layer cannot know, and a static server config cannot know either.

Hence the shape we would find most useful, in rough order of preference:

1. **Delegate.** Give `do_ReadV` an aio path like `do_Read`'s, and let the
   storage layer beneath decide whether to use it -- the OSS/cache knows whether
   it is about to serve RAM, local disk, or a remote origin.
2. **A server-side toggle**, per-export or per-oss, even a crude one. This is
   worth much more than a client-side fix to us: ROOT and XrdCl reach users
   through central builds (CERN, LCG, CMS) that they cannot change, so they are
   stuck with whatever they have. An XCache is always deployed by the site, so a
   toggle there can actually be flipped.
3. **Coarse granularity is fine** -- one offload per open file would do. That is
   precisely what the XrdCl channel workaround achieves today (one channel per
   file), and it recovered 7-9x while leaving a single file's vector reads
   serialised, which also preserves the disk-safety property above.

### Implementation notes

* The throttle already exists: `as_maxperlnk` / `as_maxpersrv`, set from
  `xrootd.async limit` and `maxtot` (`XrdXrootdConfig.cc:983-987`).
* `XrdXrootdAioFob` is already per-file, which is the natural place to express
  "one outstanding vectored read per LFN".
* `XrdXrootdAioTask::Read(long long offs, int dlen)` is single-extent, so a
  vector form needs either a new `XrdXrootdAioTask` subclass or aggregation of
  per-element completions. This looks like the real work, and is presumably part
  of why it was never done.

The client-side workaround makes this non-urgent for us. It is raised because a
7-9x factor sits behind an unused protocol field, and because rewriting filesets
is not something every analyser can be asked to do.

## A client-side cache does not have this problem at all

Dmytro Kovalskyi's **uCache** (`github.com/drkovalskyi/xrd-ucache`,
README references arXiv:2609.00400) is a per-user read cache for ROOT data over
XRootD, and it is reported to have essentially no cold-pass penalty -- its README
puts it as "the first cold run has minimal overhead, so there's little to lose in
trying it."

The reason is structural: uCache is an **XrdCl client plugin**
(`libXrdClUCache.so`), so it sits *above* the protocol layer. The client keeps
its own direct connections to the origin -- a channel per dCache pool host, and
therefore its full N-way concurrency -- while the cache is local disk beneath it.
No `kXR_readv` ever crosses an interposed server link, so the serialisation
described above simply cannot arise.

That is the point we want to make: **the penalty is not inherent to caching, it
is inherent to interposing a single-endpoint server in front of a distributed
origin.** Same workload, same origin, same client library; put the cache in the
client and the concurrency survives, put it behind an xrootd endpoint and it
collapses to one outstanding vector read.

We are not proposing XCache become a client plugin -- a site-level shared cache
serving many users is a different and necessary thing, and it is what XCache is
for. But it does mean the cost is a property of our deployment model rather than
of caching, and that is precisely what an async path for `do_ReadV` would buy
back.

(We have not read uCache's internals, so we make no claim about its chunk sizes,
readv handling or fetch concurrency -- only about where it sits in the stack,
which is what matters here.)

## Work on our side

- `BlockRun` — separating "unit of cache state" (Block) from "unit of transfer"
  (BlockRun), so a client vector read becomes one origin request instead of
  thousands. Built and measured; it is what makes the 587 number possible.
- `pfc.iosize` default raised to 1 MB, `pfc.iogap` for coalescing across gaps.
- 4 kB blocks give an 11x reduction in bytes fetched for this analysis
  (823 MB vs 9.3 GB at 1 MB blocks) — the reason for the whole exercise.
- The `NETTIME` / `CLIREAD` instrumentation above, which we would like to keep in
  some form; it made a problem we had misdiagnosed three times immediately
  obvious.

## Code

Diffs against upstream `master`, on `github.com/osschar/xrootd`:

* [`xrdcl-channels-per-host`](https://github.com/xrootd/xrootd/compare/master...osschar:xrdcl-channels-per-host) — the `ChannelsPerHost` prototype. Works,
  but it is the *wrong* answer for deployment: it only helps clients rebuilt
  against this XrdCl, and it changes the size of `XrdCl::URL`, so it breaks ABI.
  Published because it demonstrates the mechanism, not as a proposal.
* [`pfc-optimize-for-small-blocks`](https://github.com/xrootd/xrootd/compare/master...osschar:pfc-optimize-for-small-blocks) — the cache side, including the
  NETTIME/CLIREAD instrumentation these measurements came from.

## Status

Everything above is measured on `black` -> FNAL dCache (57.8 ms RTT) on
2026-09-01, with `tcpdump` on the cache's outbound leg and wall-clock
instrumentation inside XrdPfc. Capture, analysis scripts and raw logs are kept
alongside this note and can be shared.