XCache for analysis-scale reads

Running the XRootD proxy file cache with 4 kB blocks, and making it usable over a wide-area path.

XCache is the XRootD proxy file cache (XrdPfc): a site-level read cache in front of a remote origin. Two steps of work on it, and the measurements behind them.

Step 1 — Small reads

LHC analysis on thin data formats reads ROOT baskets of a few kilobytes. XCache is block-oriented, so at the historical 128 kB block size a 4 kB basket costs 128 kB of transfer. Dropping to 4 kB blocks cuts data fetched by 11x — 823 MB against 9.3 GB for the same analysis — but expands one client vector read onto thousands of cache blocks.

Four changes make that workable:

• Block runs. Separate the unit of cache state (Block) from the unit of transfer (BlockRun), so consecutive blocks are fetched as one request.
• Gap coalescing (pfc.iogap). Bridge short gaps between runs instead of splitting the request at every hole.
• Vector reads. Batch runs into one kXR_readv per client vector read: 587 origin requests instead of thousands.
• pgReadV. The same batching for the checksummed path. kXR_pgread has no vector form, so pfc.cschk net forces one request per run — 13,405 instead of 587. Proposed as kXR_pgreadv; implemented and tested.

Step 1 — XCache for small reads →

Step 2 — Analysis environment

Step 1 makes cold fetching efficient. It does not make it fast. Against FNAL dCache at 57.8 ms RTT, running the IRIS-HEP analysis grand challenge ttbar benchmark on CMS open data, a cold XCache was 8.8x slower than no cache: 249.6 s against 28.5 s direct. Warm, the same cache runs it in 11.1 s.

An XCache in production serves O(1k) clients, each with its own session, so in aggregate it is not concurrency-starved and this is invisible. It bites when a few clients each want high throughput from a cold cache — the interactive analysis case, and the case small blocks exist for.

A cache is not required to hit this. The condition is just one client, one server endpoint, several concurrent vector reads — which a multi-threaded RDF or RNTuple job over a single large file meets on its own, even with a separate TFile per thread, because the channel key is (user, host, port). Talking straight to one xrootd server or EOS does too. Behind a redirector it depends on whether the opens spread across data servers or all land on one.

Three findings:

1. do_ReadV has neither an asynchronous path nor pathID offload, so one link carries one vector read at a time. do_Read, do_PgRead and do_Write all have one. kXR_readv already carries a pathid byte that nothing in the tree reads.
2. XrdCl pools channels on (user, host, port). A client reading N files through a single cache endpoint holds one channel, and therefore one outstanding request. The benchmark runs 8 threads over 9 files, 6-7 fetched concurrently — and the measured mean outstanding origin request is 1.00.
3. Going direct to a distributed origin, the same client gets a channel per pool host. Interposing a single-endpoint cache removes exactly the parallelism the origin was providing.

The fix has two parts. Defeating channel pooling from the URL (?xrdcl.intent=) needs no new code anywhere and recovers the loss: 249.6 s to 26-37 s, parity with going direct, physics bit-identical. Better still, a server can do it unaided — redirecting each open to itself under a synthetic username puts each on its own channel, and measured that takes a cold cache from 276 s to 31 s with an entirely unmodified client, matching direct access. Properly, do_ReadV needs an asynchronous path with the decision delegated to the storage layer, since only that layer knows whether concurrency is safe — free against a WAN origin, harmful against a fully-cached file on a spinning disk.

Step 2 — Making it work in an analysis environment →

Addendum — the measurements

Packet capture on the cache's outbound leg, nanosecond instrumentation inside XrdPfc, and every hypothesis tested in turn: congestion-window collapse after idle, receive buffers, connection counts, substreams, thread pools, client concurrency. Most were wrong. Recorded because the wrong ones cost the most time.

Addendum — the measurements →

Code

Both branches sit on github.com/osschar/xrootd, shown as diffs against upstream master: pfc-optimize-for-small-blocks (the cache side — block runs, pfc.iosize, pfc.iogap, and the tracing the measurements came from) and xrdcl-channels-per-host (the client-side prototype, published to show the mechanism rather than as a proposal). The pgReadV work stacks across pgreadv (protocol and API, no wire change), pgreadv-wire (server do_PgReadV, wire, checksum retry, negotiation) and pfc-pgreadv (the cache side).

Related

XRootD is the storage framework; XCache is its XrdPfc module. uCache, by Dmytro Kovalskyi, is a per-user read cache built as an XrdCl client plugin — which is why it has no cold-pass penalty, and a useful contrast to the problem in Step 2. The benchmark is the IRIS-HEP analysis grand challenge ttbar pipeline on CMS open data.