Hadoop Migration Troubleshooting and Compatibility Reference

This page gathers the most frequent operational issues encountered during Bifrost migrations, along with the supported version matrix and a glossary of the terms used throughout the documentation. If your symptom is not listed here, consult Operations for monitoring entry points or contact [email protected].

Common Classic (Path 1) Issues

Discover fails with an API timeout

Symptom: bifrost classic discover times out while talking to the cluster manager.

Cause: The deployment export endpoint is memory-intensive on clusters with many services. The default 30-second timeout is too short.

Fix: Increase --cm-timeout to 120 seconds or more. Clusters with 100+ services commonly need this. Also check the manager's heap usage; a manager approaching heap exhaustion will produce inconsistent timeouts.

fsimage parsing fails in validate-pre but works manually

Symptom: bifrost classic validate-pre reports an fsimage parse failure, but running hdfs oiv manually against the same fsimage on the source NameNode succeeds.

Cause: The fsimage check reads the fsimage header and checksum on the source and can parse the image with hdfs oiv -p XML; the full load on isolated infrastructure runs when the engagement provides it. A parse failure usually means that (a) the fsimage was truncated during copy, (b) the parsing host has insufficient heap for the full fsimage, or (c) the isolated test installation is not the expected target version.

Fix: Compare the fsimage file size and checksum with the live NameNode. If the fsimage exceeds ~2 GB, set HADOOP_HEAPSIZE_MAX=16g on the parsing host before re-running bifrost classic validate-pre. When the full load on isolated test infrastructure is used, confirm that its Hadoop installation matches the target version (check hadoop version).

Package swap fails on some nodes

Symptom: The swap phase aborts mid-play with a package installation error on a subset of nodes.

Cause: A locked RPM database (another package manager is running), insufficient disk space in /usr, or a network issue reaching the local package mirror.

Fix: The inline rescue block automatically reverts failed nodes to the source distribution. Inspect the command output (stderr) or the file given with --log-file, and the reports in $BIFROST_STATE_DIR/<cluster>/, for the specific error. Clear the RPM database lock, free disk space, or fix the mirror and resume with bifrost classic migrate --cluster <cluster> --from-phase swap-packages --execute; Bifrost's idempotency means already-migrated nodes are skipped.

NameNode does not exit safe mode

Symptom: After the start phase, the NameNode remains in safe mode past the gate time.

Cause: Insufficient DataNodes reporting, corrupt blocks requiring manual hdfs fsck repair, or insufficient heap for processing a large fsimage.

Fix: Check hdfs dfsadmin -report to confirm DataNode count. Run hdfs fsck / to identify corrupt blocks. Increase the NameNode heap in hadoop-env.sh for unusually large fsimages. If the gate fires before the NameNode exits safe mode, Bifrost will have rolled the cluster back; investigate root cause and retry after correction. On parcel-based sources, this rollback relies on the restore point declared before the window. On a version-aligned target, the NameNode is started by the management plane takeover; investigate the same way and roll back with bifrost classic rollback --from-adopted when needed.

Common Modernize and Direct Issues

S3A writes fail with 403 Forbidden

Symptom: DistCp or Spark writes to the object storage endpoint return HTTP 403.

Cause: Incorrect credentials in core-site.xml, missing bucket, missing write permission, or (for Direct) an expired Kerberos ticket on the source HDFS.

Fix: Verify credentials in the rendered core-site.xml. Confirm the bucket exists and the user has write permission. For Kerberos sources, verify the keytab with klist -kt /var/lib/bifrost/keytabs/bifrost.keytab.

Trino returns different results than Hive

Symptom: Query parity validation fails: a query that worked on Hive returns different values on Trino.

Cause: Timezone settings, decimal precision handling, or NULL semantics differ between engines.

Fix:

• Set spark.sql.session.timeZone=UTC globally and ensure Trino sessions use UTC.
• Verify decimal precision: Hive and Trino differ in rounding behaviour; adjust column types if necessary.
• NULL handling: Hive treats empty strings as NULL in some contexts; Trino does not. Explicit NULLIF(col, '') may be required in affected queries.

Remote shuffle failures

Symptom: Spark jobs fail with shuffle read or write errors under load.

Cause: The remote shuffle workers are out of disk, under-provisioned, or unreachable from executors.

Fix:

• Check shuffle worker disk space.
• Increase the shuffle flusher buffer size for large shuffle workloads.
• Verify network connectivity between Spark executors and shuffle workers (NetworkPolicy changes can break this).
• Confirm shuffle-master HA status.

migrate-table hangs

Symptom: bifrost modernize migrate-table runs for far longer than expected and does not complete.

Cause: Insufficient executor memory for large Parquet files, S3A connection pool exhaustion, or catalog timeout.

Fix:

• Check the Spark driver logs via kubectl logs <driver-pod>.
• Increase executor memory for tables with large row groups.
• Raise fs.s3a.connection.maximum if connection pool exhaustion is suspected.
• Check catalog pod health if catalog calls are slow or failing.

Version Compatibility Matrix

The matrix applies to the Modernize and Direct target platform; Classic targets are aligned with the source and are listed in the Classic support matrix. Bifrost ships with tested combinations of the following components. Using versions outside these ranges is not supported.

Component	Version	Iceberg version	Notes
Apache Spark	3.5.x	1.5.x, 1.6.x, 1.7.x	Spark 3.5 is the default supported line.
Apache Spark	4.0.x	1.10.x or later	Spark 4.0 requires the iceberg-spark-runtime-4.0_2.13 artifact, first published in Iceberg 1.10.0.
Trino	477 to 480	1.5.x, 1.6.x, 1.7.x	Iceberg connector bundled; REST catalog supported.
Apache Polaris	1.3.0	Iceberg REST spec v1
Apache Gravitino	0.8.x	Iceberg REST spec v1	Exposes Iceberg REST as a subset of its API.
Apache Iceberg	1.5.2, 1.6.1, 1.10.0+	N/A	Pick the Iceberg release compatible with the chosen Spark line.
Object storage (Ceph)	Rook 1.15.5+ for Squid (Ceph 19.2); Rook 1.17 supports Reef and Squid; Rook 1.18-1.19 support Reef and Squid (Quincy dropped)	N/A	Squid support landed mid-1.15 series; verify against Rook release notes for the exact patch version.
Apache Airflow	3.0.x	N/A	KubernetesExecutor.
Apache Celeborn	0.5.x	N/A	Supports Spark 2.4 through 3.5. For Spark 4.x, use Celeborn 0.6.x (which introduced Spark 4.0 support).
YuniKorn	1.5.x or 1.6.x	N/A	Gang scheduling, hierarchical queues.
Spark Operator	2.1.x or 2.2.x (for Spark 3.5); 2.3.x (for Spark 4.0)	N/A	Each Spark Operator release is pinned 1:1 to a base Spark version.
Ilum	6.3.x	N/A	Supports Spark 3.5 and 4.x.

Critical version constraints

• Spark with Iceberg. Spark 3.5 requires iceberg-spark-runtime-3.5_2.12. Spark 4.0 requires iceberg-spark-runtime-4.0_2.13, which first appeared in Iceberg 1.10.0. Mixing Spark and Iceberg-runtime artifact versions causes NoSuchMethodError at class-load time.
• Trino with credential vending. Trino's iceberg.rest-catalog.vended-credentials-enabled=true is supported by Trino releases that ship the full Iceberg REST catalog OAuth2 property set. The tested range is Trino 477-480 (see the compatibility matrix above); production deployments should run the most recent 48x release to pick up security fixes. Verify against the active Trino security advisories for the Trino version in use. Azure vended credentials are not yet supported.
• Celeborn with Spark. Celeborn 0.5.x supports Spark 2.4 through 3.5. Spark 4.0 and 4.1 require Celeborn 0.6.x. Set spark.shuffle.manager=org.apache.celeborn.client.spark.CelebornShuffleManager.
• Iceberg table format version. Use v2 tables for row-level deletes (DELETE, UPDATE, MERGE). Trino 411 or later and Spark 3.4 or later support v2 writes; v1 tables are read-only for delete operations.
• Spark Operator with Spark. Each Spark Operator minor release is bound to a specific Spark base image. Do not mix Spark Operator 2.1.x/2.2.x with a Spark 4.0 image, or Spark Operator 2.3.x with a Spark 3.5 image.

Wave-Level Failure Semantics

Recap of how Modernize and Direct handle wave failures. The full discussion is in Validation and rollback.

• A failed table is quarantined (marked FAILED), not the whole wave.
• Other tables in the wave continue normally.
• A wave is marked COMPLETE only when every non-quarantined table passes validation.
• Quarantined tables do not block wave completion but must be resolved before decommission.
• Retry policy is 3 attempts with exponential back-off (5 minutes, 15 minutes, 45 minutes).
• After 3 failures, manual intervention is required.

CI Integration

Every Bifrost command is suitable for CI pipeline use. Exit codes follow a stable contract:

Code	Meaning
0	Success: PROCEED, or WARN without --strict.
1	Generic failure (configuration error, unexpected exception, invalid usage).
2	ABORT verdict from the decision engine, or a refusal at an irreversible step; automated rollback triggered where applicable.
3	WARN verdict under --strict.
4	The source cluster could not be reached.
5	A rollback ran but did not complete.
64	The command is not available in this release.

Wrap Bifrost commands in a script that distinguishes codes 2 and 3 from 1 to route notifications correctly. --no-notify suppresses Slack and IT service management channels for dry-runs and local validation.

Getting Help

• Ilum support: customers with Ilum Enterprise subscriptions have access to support channels documented in Help and support.
• Email: [email protected] for commercial inquiries and migration program support.
• Ilum documentation: see the overview, architecture, security, and production deployment pages for details on the destination platform.

Glossary

Term	Definition
AQE	Adaptive Query Execution. Spark 3.x feature that dynamically adjusts query plans at runtime based on actual data statistics (partition coalescing, skew handling).
Canary	A before-and-after comparison of cluster state and behaviour. Signatures recorded on the source before the window are compared with the same signatures after the management plane takeover.
Celeborn	The remote shuffle service used on the target platform. Decouples shuffle data from Spark executor lifetime, enabling safe Dynamic Resource Allocation on Kubernetes.
Configuration plane	The desired configuration that the management plane holds. The takeover certifies it, validate-post names later changes, and restore-config-plane reverts them.
CRD	Custom Resource Definition. A Kubernetes extension mechanism that allows defining custom object types (for example, SparkApplication, TableMigration).
Credential vending	The practice of an Iceberg catalog generating temporary, scoped storage credentials per request, eliminating long-lived storage keys.
Day-2 drill	An acceptance drill (day2-drill) that exercises the day-2 operations of the management plane on the adopted cluster before go-live, each bracketed by canary captures.
Decision engine	The component of Bifrost that evaluates automated go/no-go checks at each phase gate and returns PROCEED, WARN, or ABORT.
DistCp	Distributed Copy. Tool for large-scale parallel data movement between file systems.
DRA	Dynamic Resource Allocation. Spark's ability to add and remove executors during job execution based on workload demand.
Dual-read bridge	The strangler-facade configuration during Modernize that exposes both legacy and migrated tables through a single Trino interface.
Erasure coding (EC)	A data protection method that splits data into fragments and generates parity fragments. EC(4+2) means 4 data + 2 parity fragments, providing 50 % raw-capacity overhead (vs. 200 % for triple replication).
ESO (External Secrets Operator)	A Kubernetes operator that synchronizes secrets from external stores (HashiCorp Vault, Azure Key Vault, AWS Secrets Manager, GCP Secret Manager, and others) into native Kubernetes Secret resources.
Exchange manager	Trino's mechanism for spilling intermediate data to external storage during fault-tolerant execution, enabling task-level retries.
Gang scheduling	A scheduling pattern in which all pods of a job (driver + executors) are scheduled atomically. Prevents deadlocks where a driver is scheduled but no executors can be placed.
HA roll plan	The shared rolling-restart order used by Bifrost day-2 commands: dependency order, standby before active, with a health gate after every step.
Ilum Cluster Ops	The Ilum interface for operating a Hadoop cluster managed by Apache Ambari (Ilum build) after a Classic migration.
JuiceFS	A distributed file system that exposes object storage through a native HDFS SDK. Used as a compatibility bridge for applications that require HDFS API semantics against an S3-backed store.
Magic committer	An S3A committer that uses multipart upload to write data directly to its final location, eliminating the rename-as-copy pathology that plagues naive S3 writers.
Management plane takeover	Phase 9 of a Classic migration. The migrated cluster is adopted by Apache Ambari (Ilum build), which replaces Cloudera Manager and starts the services.
Mirror lock	Locking the local package mirror on every node on one release of a module (mirror-lock) before patch rolls it.
OPA	Open Policy Agent. The policy engine used by Trino on the target platform for fine-grained SQL authorization.
Plan validation	An offline check of the migration plan (validate-plan) for completeness, configuration drift, and endpoint changes before any destructive step.
Restore point	An LVM or VM snapshot of the NameNode metadata volume, declared before the window for parcel-based sources and checked by validate-pre.
Silence period	A configurable window (default 30 days) during which Bifrost verifies that no production reads or writes touch a legacy service before it is decommissioned.
Strangler facade	A migration pattern in which a facade (Trino table redirection) gradually routes traffic from legacy systems to modern replacements, allowing incremental migration without breaking consumers.
Table redirection	A Trino feature in which queries to one catalog are transparently redirected to another catalog for specific tables, enabling gradual migration without changing SQL.
Version-aligned target	The default Classic target, whose component versions are aligned with the source, so that no HDFS layout upgrade is needed.
Wave	A group of tables that migrate together, with internal parallelism, as a single unit of Modernize execution.
YuniKorn	The Kubernetes batch scheduler used on the target platform for YARN-style hierarchical queue management, fair scheduling, and gang scheduling.

Next Steps

• Overview: Bifrost at a glance.
• CLI reference: every command.
• Operations: monitoring, backup, capacity, readiness.