Step 2 — Making it work in an analysis environment

Concurrent kXR_readv.

For: Andy Hanushevsky From: Matevž Tadel (with analysis by Claude, working in the xrootd tree) Date: 2026-09-02 Depends on: Step 1 (xcache-step1-small-reads.md), which stands as written.

Step 1 makes small-block caching efficient. It does not make it fast enough to use. This document covers what is needed for that, and adds a second request to the one Step 1 made.

What Step 1 asked for, in one line

kXR_pgreadv: N (offset, length) pairs like kXR_readv, crc32c per 4 kB page like kXR_pgread, because pfc.cschk net currently forces XCache into one origin request per cache block and there is no vector form to batch them into.

That ask is unchanged, and Step 1's implementation notes still hold — it is built and working on branch pfc-pgreadv.

Measurements over a wide-area path

Step 1's numbers came from a UCSD origin at 8 ms RTT, where per-request cost is small enough that request count looks like an efficiency question. We have since run the IRIS-HEP AGC ttbar analysis (ROOT RDataFrame, TTreeCache, CMS open data NanoAODv9) against FNAL dCache at 57.8 ms RTT, with tcpdump on the cache's outbound leg and nanosecond instrumentation inside XrdPfc.

Identical physics, identical client, same host and link:

The cold number is the problem, and it is not what we assumed.

The network is not the bottleneck

From the packet capture, per origin response:

Think time is one RTT and is identical. dCache is not slow. The wire is slightly faster for the cache than for ROOT. The entire difference was idle time.

We found a real cause for part of it — the cache's pool connections idle ~2.1 s between requests, past the minimum RTO, so the origin's congestion window resets to IW10 (measured first round: 14 kB, exactly 10 x 1428 MSS) and each ~1 MB response spends ~10 RTTs ramping. Raising client concurrency warmed the connections and fixed it completely: transfer time per response fell from 336 ms to 60.8 ms, versus ROOT direct's 58.9 ms — parity.

The run got slower. 245 s instead of 229 s.

The actual bottleneck: no request overlap through a cache

The benchmark runs 8 RDataFrame threads over 9 input files, and the cache confirms it has them all: 9 distinct IO objects for 9 distinct files, 6-7 of them being fetched concurrently, peak 9. There is no shortage of independent work to overlap.

Instrumenting every remote request with wall-clock issue/completion timestamps:

MEAN remote requests in flight : 1.00      peak 8
MEAN client reads in flight    : 0.99      peak 8

Exactly 1.00, across three configurations with 787 / 787 / 1347 requests at median latencies of 393 / 286 / 168 ms. The run is n_requests x latency / 1, which is why halving latency bought nothing at all.

The client-side counter is the telling one. In the cold run there were 787 client vector reads and 787 origin requests — 1:1, and the cache logged one login. ROOT opens a single session to the cache, all nine files ride it, and vector reads on it do not overlap. Going direct, the same ROOT process gets twelve links to twelve pool nodes and overlaps freely.

The arithmetic then closes on every measurement we have:

The rule, stated plainly and confirmed by the split-process test below: a client's outstanding-request concurrency equals the number of server sessions it holds, and nothing else. Going direct to dCache, ROOT gets a session per pool node — twelve of them — and overlaps across them. An XCache is a single endpoint, so the same client gets one session and one outstanding request.

This is not a tuning problem, it is structural: interposing any single-endpoint cache in front of a distributed origin collapses exactly the parallelism the origin's own architecture was providing. Warm, this is invisible (14 ms x 787 = 11 s, still beating direct). Cold, over a WAN, it is the whole story.

Effect on the pgReadV request

If round trips are serialized, their count is the only thing that matters, and that is precisely what pgReadV controls. On this workload, at pfc.blocksize 4k and pfc.iosize 1m:

23x more round trips. At the measured 282 ms per serialized round trip that is not "somewhat slower", it is roughly an hour instead of three minutes. Without kXR_pgreadv, end-to-end checksums are not merely expensive on a WAN cache at small block size — they are unusable. Step 1 argued this on efficiency grounds; the WAN data turns it into a feasibility argument.

The second ask: make the async path reachable for do_ReadV

The cause is in the server. Making vector reads asynchronous unconditionally would be wrong, for the reason that presumably kept them synchronous. The request is narrower: make the async path reachable, and let the layer that knows the storage decide whether to take it.

What we found

ClientReadVRequest in XProtocol.hh already carries a pathid byte:

struct ClientReadVRequest {
   kXR_char  streamid[2];
   kXR_unt16 requestid;
   kXR_char  reserved[15];
   kXR_char  pathid;      // on the wire, referenced nowhere in the tree
   kXR_int32 dlen;
};

Nothing in the tree reads it -- not server, not client. Meanwhile:

do_ReadV therefore completes synchronously on the link's thread, so one link carries one vector read at a time. Three measurements pin it there:

• Not ROOT. A bare XrdCl probe issuing K concurrent vector reads on one session: K=1 gives 78 ms/req over 30 requests; K=8 gives 93 ms/req. No overlap.
• Not pss / posix / pfc. Instrumenting the cache's client-facing reads, which sit above all three: MEAN 0.98, peak 1. The cache never saw two client vector reads at once while eight were outstanding.
• Not XrdCl. Loopback capture, K=8: eight readv requests pipelined onto one socket inside 100 us, before any response; first response 860 ms later. XrdCl delivered the concurrency; the server did not use it.

This also explains why SubStreamsPerChannel does nothing for a readv workload (we measured 2 -> 16 connections, no change). Substreams are parallel paths; read, pgread and write all use them, and readv discards the field.

Note for pgReadV: because do_PgRead already honours pathID and offloads, a do_PgReadV modelled on it inherits the right behaviour for free. The problem is specific to plain kXR_readv.

It is not only a cache problem

An XCache is a guaranteed instance of this, not the only one. The condition is just one client, one server endpoint, several concurrent vector reads, and that is easy to meet with no cache in sight:

• A multi-threaded RDF or RNTuple job over a single large file. Threads work on different parts of the file, and each may open its own TFile — but XrdCl keys channels on (user, host, port), so every one of those opens lands on the same channel. Measured directly: eight XrdCl::File handles on the same path from one process share two sockets and gain nothing from concurrency (5.13 s); forcing them onto separate channels gives sixteen sockets and 3.82 s.
• A job talking straight to one xrootd server, EOS included.
• Through a redirector it depends. If the opens are spread across data servers, the client gets a channel per server and the limit stays hidden. If they all land on the same server — a single replica, or a redirector that pins them — it appears exactly as it does in front of a cache.

The ?xrdcl.intent= workaround applies unchanged in every one of these: it is a property of the URL, not of the cache.

A server-side way to split channels

Andy pointed out that a server can hand a connecting client a different username, which is enough to make XrdCl open a new socket. Traced through the code, it works, and it is better than anything client-side:

1. An OSS/OFS plugin returns SFS_REDIRECT with the ErrInfo code as the port and the message as the host (XrdSfsInterface.hh:115).
2. That reaches the client as kXR_redirect, message passed through (XrdXrootdXeq.cc:3936).
3. XrdCl builds a URL from it, and URL::ParseHostInfo accepts user[:pass]@host.
4. URL::ComputeHostId() puts user@ into pHostId, and GetChannelId() is protocol://pHostId/ -- so the redirect lands on a new channel.

So a cache can redirect each open to itself under a synthetic username and get one channel per open, with no client change, no protocol change, and nothing for the user to do. That is strictly better than asking analysers to rewrite their filesets with ?xrdcl.intent=.

Note that XrdCl inherits the client's original username only when the redirect omits one, so a redirect-supplied name wins.

Measured

Tested on 2026-09-03 with a bare redirector on one port bouncing read opens to the cache on another, and the client fileset left completely unmodified. The static xrootd.redirect directive cannot express a username -- xred_php() validates the target with XrdNetAddr::Set(), so u1@localhost is rejected -- so the rotation was done with a throwaway patch at the redirect send.

787 remote requests either way, so the work is identical. ROOT direct to FNAL is 28.5 s: the redirect brings a cold cache to parity with not caching at all, and asks nothing of the client. That is a ~9x improvement from a server change alone, and it beats both URL-level workarounds (xrdcl.intent gave 47.8 s, a distinct username per file 26-37 s).

In flight settles at 2.83 rather than 8 because per-response latency falls once requests overlap -- the connections stay warm, so the congestion window does not collapse between requests. 787 requests at ~113 ms over 2.83 concurrent is the 31 s.

Where it has to be raised, and one thing it runs into

The plumbing to raise this inside the cache -- rather than from a separate redirector -- already exists, and needs no protocol change: XrdPssFile::Open puts a full URL into the open env and returns -EDESTADDRREQ; XrdOfs.cc:788 turns that into SFS_REDIRECT with ecode = -1; the protocol layer sends the URL as the redirect body; XrdCl parses user@host out of it, and URL::GetChannelId() includes the user. XrdPssFile::Open already has Env.secEnv() in hand, so the loop guard is a comparison -- though it has to look at tident, not name: under sec.protocol host there is no authenticated name.

But a self-redirect to the same host:port differing only in username loops on a stock client. XRootDMsgHandler::RetryAtServer gates its "new server" branch on URL::GetLocation(), which is protocol://host:port/path and deliberately omits the user. The redirect therefore looks like the same server, the request is retried on the existing channel, the server redirects again, and the client hits its redirect limit. Comparing GetChannelId() as well is a one-line fix, and with a patched client the open succeeds and the channels appear.

Avoiding that client change is possible: redirect to a second port of the same instance (xrd.protocol xroot:<p> +port lets one process listen on two ports, so GetLocation() differs and a stock client takes the new-server branch). Measured with a native XrdCl client, this works well. Six distinct paths, 36 vector reads, concurrency 6:

8.5x, entirely server-side. So the mechanism is sound. Two limits found:

• Repeated opens of the same path get one channel. Eight opens of one path through the redirect produced a single synthetic user: XrdCl resolves the redirect once and reuses that endpoint for the rest. So this does not help the single-large-file case either -- see below.
• With ROOT, the redirect target must be a different process. Full matrix, all runs complete, cold AGC against FNAL:

So the pss-layer hook is the right place and the mechanism is sound: two processes work with either mechanism. What fails is ROOT against a self-redirecting single instance. The client cannot be the cause on its own, since it sees byte-identical URLs in both topologies, and a native XrdCl client does benefit even in the single-instance case. That points at the one process serving both the initial and the redirected links, and it is the remaining question.

Not ROOT's redirect handling, incidentally: TNetXNGFile::Open only inspects errRedirect on failure (storing fNewUrl for TFile to retry), so a redirect that XrdCl follows successfully is invisible to ROOT.

So the measured 276 s -> 31 s above stands, but it was obtained by redirecting to a genuinely different endpoint. Whether the same-endpoint form can be made to work without a client change is open.

Still worth measuring before production: the extra round trip per open; authentication running again per channel, free under sec.protocol host but not under GSI or tokens; and monitoring, mapping and quota all seeing synthetic usernames. A real implementation belongs in the OSS/OFS layer returning SFS_REDIRECT, where it can vary per open and be switched off, rather than in the protocol layer's static route table.

XrdSsi already does this, client-side

Andy pointed at XrdSsi, and it is the same trick, done in the right place. XrdSsiServReal::GenURL builds the endpoint URL with a numeric synthetic username:

if (uEnt == 0) xUsr = xAt = "";
   else {snprintf(uBuff, sizeof(uBuff), "%d", uEnt);
         xUsr = uBuff; xAt = "@";
        }

giving root://4@host:port/path, purely so that sessions spread across channels. XrdSsiScale manages the spread and answers the "how many?" question that a fixed setting cannot: defSprd = 4, maxSprd = 1024, with auto-tuning driven by per-channel pending counts (minTune, midTune, maxTune, quadratic then linear growth).

The important part is where: SSI decorates the URL client-side, at open. That is why it works, and it is what our fileset rewrite was doing by hand. Measured in SSI's exact form against a single cache on a single port, with an unmodified ROOT:

No client change of any kind: the server log shows zero redirects, and ROOT loads /usr/lib64/libXrdCl.so.3, so none of our XrdCl work is even in the process. Whether our RetryAtServer change is present is therefore irrelevant on this path -- it is only reached on a redirect, and there is none.

Only six distinct users appeared because the index assignment collided across JSON entries, which incidentally shows the gain scales with the number of distinct channels: nine users gave 26-37 s. That is precisely what XrdSsiScale exists to tune, and an argument for adopting its logic rather than a fixed setting.

Our redirect experiments were trying to achieve server-side what SSI achieves client-side, which is the wrong end.

So the shape worth building is a small URL decorator, either in the RDF/RNTuple data source or inside XrdCl so that nothing above it needs to care -- with XrdSsiScale as the model for the tuning rather than a fixed N.

The login username is not visible to the storage layer

Worth recording because it constrains any server-side variant. The login name from kXR_login is sanitised and then handed only to Link->setID(uname, pid) (XrdXrootdXeq.cc:1100). Nothing assigns it to XrdSecEntity::name, which is set by authentication protocols and is null under sec.protocol host. XrdSecEntity::tident is documented as //!< Trace identifier always preset -- it merely happens to begin with the login name, because that is what the link ID is built from.

So an OSS/pss-level hook cannot read the client's login name through any documented field. Our first loop guard read tident and worked by accident; a real one has to key on something explicit, such as a CGI marker placed in the redirect URL, which does reach the storage layer -- XrdOfs builds the open env as XrdOucEnv Open_Env(info, 0, client) from the client's opaque info.

Scope: this is the few-client case

An XCache in production serves O(1k) clients, each with its own session and its own outstanding vector read. In aggregate the server is not concurrency-starved, so the serialisation costs little there. It is a known limitation; what is new here is the size of it in the opposite regime.

It bites when a small number of clients each want high throughput from a cold cache. That is the interactive analysis case, and it is exactly the case small block sizes exist for. The quantity that matters here is per-client latency, not server aggregate throughput.

Why it hurts caches

XrdCl keys channels on (user, host, port). A client reading N files from one XCache gets one channel, hence one outstanding vector read. The same client going direct to dCache has its N files spread over N pool hosts, so it gets N channels and N-way concurrency. Interposing a single-endpoint cache removes exactly the parallelism the distributed origin was supplying.

Full AGC cold run, FNAL dCache at 57.8 ms RTT:

Physics bit-identical throughout. So a 7-9x recovery from nothing but defeating channel pooling -- the cold cache reaches parity with direct.

Why we think the decision belongs below the protocol

We assume readv was left synchronous deliberately. Throwing N concurrent vector reads at a classic xrootd data server is a bad idea, especially several on the same file, when that file lives on one spinning disk -- the seek pattern is exactly what a vectored read is supposed to avoid. That reasoning is sound and we are not asking you to discard it.

But it is a property of the storage, not of the request. For a cache in front of remote multi-point storage it does not apply at all. And the cache cannot answer it once and for all either: the same XCache serves a cold file from a WAN origin (concurrency is free and worth ~8x) and a fully-cached file from a local spinning disk (concurrency is harmful) -- sometimes within one request. So the protocol layer cannot know, and a static server config cannot know either.

Hence the shape we would find most useful, in rough order of preference:

1. Delegate. Give do_ReadV an aio path like do_Read's, and let the storage layer beneath decide whether to use it -- the OSS/cache knows whether it is about to serve RAM, local disk, or a remote origin.
2. A server-side toggle, per-export or per-oss, even a crude one. This is worth much more than a client-side fix to us: ROOT and XrdCl reach users through central builds (CERN, LCG, CMS) that they cannot change, so they are stuck with whatever they have. An XCache is always deployed by the site, so a toggle there can actually be flipped.
3. Coarse granularity is fine -- one offload per open file would do. That is precisely what the XrdCl channel workaround achieves today (one channel per file), and it recovered 7-9x while leaving a single file's vector reads serialised, which also preserves the disk-safety property above.

Implementation notes

• The throttle already exists: as_maxperlnk / as_maxpersrv, set from xrootd.async limit and maxtot (XrdXrootdConfig.cc:983-987).
• XrdXrootdAioFob is already per-file, which is the natural place to express "one outstanding vectored read per LFN".
• XrdXrootdAioTask::Read(long long offs, int dlen) is single-extent, so a vector form needs either a new XrdXrootdAioTask subclass or aggregation of per-element completions. This looks like the real work, and is presumably part of why it was never done.

The client-side workaround makes this non-urgent for us. It is raised because a 7-9x factor sits behind an unused protocol field, and because rewriting filesets is not something every analyser can be asked to do.

A client-side cache does not have this problem at all

Dmytro Kovalskyi's uCache (github.com/drkovalskyi/xrd-ucache, README references arXiv:2609.00400) is a per-user read cache for ROOT data over XRootD, and it is reported to have essentially no cold-pass penalty -- its README puts it as "the first cold run has minimal overhead, so there's little to lose in trying it."

The reason is structural: uCache is an XrdCl client plugin (libXrdClUCache.so), so it sits above the protocol layer. The client keeps its own direct connections to the origin -- a channel per dCache pool host, and therefore its full N-way concurrency -- while the cache is local disk beneath it. No kXR_readv ever crosses an interposed server link, so the serialisation described above simply cannot arise.

That is the point we want to make: the penalty is not inherent to caching, it is inherent to interposing a single-endpoint server in front of a distributed origin. Same workload, same origin, same client library; put the cache in the client and the concurrency survives, put it behind an xrootd endpoint and it collapses to one outstanding vector read.

We are not proposing XCache become a client plugin -- a site-level shared cache serving many users is a different and necessary thing, and it is what XCache is for. But it does mean the cost is a property of our deployment model rather than of caching, and that is precisely what an async path for do_ReadV would buy back.

(We have not read uCache's internals, so we make no claim about its chunk sizes, readv handling or fetch concurrency -- only about where it sits in the stack, which is what matters here.)

Work on our side

• BlockRun — separating "unit of cache state" (Block) from "unit of transfer" (BlockRun), so a client vector read becomes one origin request instead of thousands. Built and measured; it is what makes the 587 number possible.
• pfc.iosize default raised to 1 MB, pfc.iogap for coalescing across gaps.
• 4 kB blocks give an 11x reduction in bytes fetched for this analysis (823 MB vs 9.3 GB at 1 MB blocks) — the reason for the whole exercise.
• The NETTIME / CLIREAD instrumentation above, which we would like to keep in some form; it made a problem we had misdiagnosed three times immediately obvious.

Code

Diffs against upstream master, on github.com/osschar/xrootd:

• xrdcl-channels-per-host — the ChannelsPerHost prototype. Works, but it is the wrong answer for deployment: it only helps clients rebuilt against this XrdCl, and it changes the size of XrdCl::URL, so it breaks ABI. Published because it demonstrates the mechanism, not as a proposal.
• pfc-optimize-for-small-blocks — the cache side, including the NETTIME/CLIREAD instrumentation these measurements came from.

Status

Everything above is measured on black -> FNAL dCache (57.8 ms RTT) on 2026-09-01, with tcpdump on the cache's outbound leg and wall-clock instrumentation inside XrdPfc. Capture, analysis scripts and raw logs are kept alongside this note and can be shared.