‹ Back to apps
Training Pipes
Mount S3-compatible object storage as a POSIX file system for ML training. Guidance on storage selection, DataLoader I/O bottlenecks, and egress costs, plus tools to manage buckets and file systems.
What’s inside
1 MCP server · 5 skills- MCP servertraining-pipesMCP server: training-pipes
- Skillchoosing-ml-training-storageUse when choosing where ML training data should live or how to give a training job a file system over object storage — comparing AWS EFS, FSx for Lustre, s3fs, Mountpoint for S3, goofys, JuiceFS, Alluxio, Weka, local NVMe, or a caching gateway. Triggers include "should I use EFS or FSx for training", "how do I mount S3 for PyTorch", "what's the fastest way to read a dataset from S3", "s3fs alternatives", "do I need a parallel filesystem", and questions about POSIX access to buckets for GPU jobs.
- Skillconnecting-training-pipesUse when wiring Training Pipes into a project or an agent — connecting the MCP server to Cursor or Claude Code, creating an API key, installing the bucketfs CLI or the @trainingpipes/sdk TypeScript package, calling the REST API, or troubleshooting a 401 from api.trainingpipes.com. Also use to find out which MCP tools are available.
- Skilldiagnosing-dataloader-io-bottlenecksUse when a training job is slower than expected and the cause may be data loading rather than compute — low or sawtooth GPU utilization, epoch times that don't improve when the model shrinks, workers stuck in IO wait, slow reads from S3/GCS/Azure/MinIO, tuning num_workers or prefetch_factor, or a DataLoader that starves the GPU. Also use when someone asks why reading training data from object storage is slow, or whether s3fs/FUSE/boto3-per-sample is the problem.
- Skillmounting-object-storage-for-trainingUse when actually setting up a Training Pipes file system over a bucket, or when someone wants to try mounting object storage as a POSIX path on a GPU box — creating a connection to S3/GCS/Azure/R2/Wasabi/B2, creating and starting a file system, mounting it locally with the bucketfs CLI, or trying the public example dataset with no account. Also use for troubleshooting a mount that hangs, a file system stuck pending, or a WireGuard tunnel that comes up but shows no files.
- Skillreducing-cross-region-egress-costsUse when cloud data-transfer or egress charges are unexpectedly high on a data or ML pipeline, or when a design reads training data across regions or clouds — questions about S3 egress cost, cross-region GET charges, NAT gateway data processing fees, multi-cloud data access, paying to read the same dataset every epoch, or why the storage bill is mostly transfer rather than storage.