Diagnosing Cloud Mac Build I/O Spikes from Spotlight and File Watchers

The same commit is built three times in a row on a cloud Mac: the first build completes normally, the second suddenly slows down, and the third returns to normal. CPU usage never remains saturated, and network downloads finished long ago. This kind of variability is often misdiagnosed as insufficient hardware. In practice, Spotlight may be scanning DerivedData, an editor may be recursively watching the entire workspace, or several CI jobs may be writing to the same cache directory at once. The key is not to clear every cache first, but to preserve the evidence, quantify I/O activity, and narrow down each source of contention in turn.

Establish a comparable build baseline

Before troubleshooting, pin the commit, Xcode version, build target, and cache paths. Do not mix the initial dependency download with subsequent incremental builds in the same dataset, and do not run cleanup scripts while a build is in progress.

set -o pipefail

WORKSPACE="$HOME/ci/work/app"
DERIVED="$HOME/ci/derived/app-main"
PACKAGES="$HOME/ci/packages/app"

mkdir -p "$DERIVED" "$PACKAGES"
cd "$WORKSPACE"

xcodebuild \
  -resolvePackageDependencies \
  -clonedSourcePackagesDirPath "$PACKAGES"

for run in 1 2 3; do
  /usr/bin/time -lp xcodebuild \
    -workspace App.xcworkspace \
    -scheme App \
    -configuration Debug \
    -destination 'generic/platform=iOS Simulator' \
    -derivedDataPath "$DERIVED" \
    -clonedSourcePackagesDirPath "$PACKAGES" \
    build 2>&1 | tee "$HOME/ci/build-$run.log"
done

Record wall time, maximum resident memory, the dependency-resolution phase in the logs, and whether any other jobs were active when the slowdown occurred. If all three runs are slow, the project or toolchain is usually the first place to look. If only individual runs are abnormal, background I/O contention is more likely.

Capture I/O activity with system tools

Check the indexing status first, then capture file access during an abnormal build. fs_usage requires administrator privileges and produces a large volume of output, so restrict it to the relevant processes before filtering by directory keywords.

mdutil -s /

sudo fs_usage -w -f filesys mds mdworker_shared |
  grep -E 'DerivedData|SourcePackages|/ci/work/'

In another terminal, inspect the related processes:

ps -axo pid,ppid,%cpu,%mem,etime,command |
  grep -E 'mds|mdworker|xcodebuild|swift-frontend|SourceKit' |
  grep -v grep

If mdworker only reads newly checked-out source files briefly, that alone does not prove it is the root cause. Stronger evidence is sustained scanning of large numbers of intermediate artifacts during the slowdown, with the scanned paths overlapping the paths being written by the build.

Do not run rm -rf DerivedData before collecting evidence. Clearing the cache may make the next build slower and erase the clues needed to determine which process repeatedly accessed which files.

Narrow Spotlight’s indexing scope

Disabling indexing for the system volume is not recommended. Developers may still depend on system search, and a global change can hide the directory layout that is actually causing the problem. A safer approach is to place frequently generated, fully rebuildable caches on a separate APFS volume and adjust indexing only for that volume.

After confirming the cache volume’s mount point, run:

mdutil -s /Volumes/CICache
sudo mdutil -i off /Volumes/CICache
mdutil -s /Volumes/CICache

Keep indexing enabled for source code, documentation, and other material that needs to remain searchable. Put DerivedData, package download caches, test attachments, and temporary archive directories on the cache volume. To restore indexing later:

sudo mdutil -i on /Volumes/CICache
sudo mdutil -E /Volumes/CICache

Do not place signing materials, long-lived artifacts, and temporary caches within the same cleanup boundary. Excluding a volume from indexing addresses scan contention only; it does not replace access controls or data classification.

Control recursive file watchers and shared caches

Editors, code generators, and development servers often watch the repository root. If their watch rules include .git, DerivedData, test attachments, or package caches, every build can trigger tens of thousands of unnecessary events.

Reduce the watch scope

Limit watch targets to source and configuration directories, and explicitly exclude the following:

• .git and temporary files created during checkout;
• DerivedData, archives, and test result directories;
• SourcePackages and other rebuildable dependency caches;
• logs, coverage reports, and generated code output.

First quit the suspected editor or agent, then run the same baseline again. If the variability disappears, restore the processes one at a time. This makes it easier to identify the responsible process than changing every tool at once.

Give parallel jobs separate write paths

Sequential builds can reuse a project cache, but parallel CI jobs should not write to the same DerivedData directory. At a minimum, the path should include repository and job identifiers:

SAFE_REPO="${REPO_NAME//[^a-zA-Z0-9_-]/_}"
SAFE_JOB="${JOB_ID//[^a-zA-Z0-9_-]/_}"

export DERIVED_DATA="$HOME/ci/derived/$SAFE_REPO/$SAFE_JOB"
export PACKAGE_CACHE="$HOME/ci/packages/$SAFE_REPO"

mkdir -p "$DERIVED_DATA" "$PACKAGE_CACHE"

A package cache can be shared for reads, but dependency updates may still create write contention. In highly concurrent environments, resolve dependencies first in a controlled step, then have the build jobs use the stable resolution result.

Keep cleanup and regression validation separate

Cleanup scripts should avoid any running xcodebuild process and remove expired caches by job directory instead of unconditionally clearing the top-level directory every day. Start by listing candidates only:

DERIVED_ROOT="$HOME/ci/derived"

if ! pgrep -x xcodebuild >/dev/null; then
  find "$DERIVED_ROOT" \
    -mindepth 2 -maxdepth 2 \
    -type d -mtime +7 -print
fi

After confirming the directory depth and retention period, move the deletion step into a separate job. For validation, rerun the fixed baseline three times while monitoring fs_usage. Success does not mean that one run happens to be exceptionally fast. It means indexing processes no longer scan build directories continuously, parallel jobs do not share write paths, and the distribution of build phases remains consistent across consecutive runs.

Finally, record the Xcode version, commit, DerivedData path, package cache path, active watcher processes, and indexing status in the build diagnostics. The next time performance varies, you can begin by comparing environment differences instead of clearing caches and starting the guesswork over again.

Frequently asked questions