Hadoop Migration: First Discovery Run

This page walks through the first hours of working with Bifrost: confirming prerequisites, obtaining the distribution, installing it on a controller host, understanding the configuration-as-data model, and running a first discovery pass against a non-production cluster.

Prerequisites

Bifrost runs on a dedicated controller host that has network access to the source cluster and, for Modernize and Direct paths, to the target Kubernetes cluster. The controller host does not need to be part of either cluster.

Controller host

• Operating system: Red Hat Enterprise Linux 8, 9, or 10, or a compatible distribution (Rocky Linux, AlmaLinux, Oracle Linux), or an Ubuntu or Debian-based distribution, on x86_64. See the Support matrix.
• Python 3.10, 3.11, or 3.12. The installer places every other dependency, including Ansible, in a local virtual environment inside the Bifrost directory; nothing is installed system-wide, and neither root access nor a JDK is needed on the controller.
• sha256sum, used to verify the delivered bundle.
• SSH access to every node in the source cluster with passwordless sudo, using public-key or GSSAPI (Kerberos) authentication. On clusters that accept only GSSAPI, a Kerberos client and a valid ticket are needed on the controller.
• A file creation mask (umask) of 022, so that the inventory stays readable for other operators.
• A state directory for run records, reports, and rollback manifests: $BIFROST_STATE_DIR, by default /var/lib/bifrost/state. Set it explicitly when Bifrost does not run as root.

Source environment

• Credentials for the source cluster management plane. For Classic discovery, a read-only Cloudera Manager API account is sufficient, together with the CA bundle that verifies the Cloudera Manager certificate.
• On a Kerberized source, a technical account with a keytab for discovery, and the HDFS superuser keytab used to list encryption zones.
• On a source with HDFS encryption zones, a Ranger KMS key administrator identity used to read the key versions.
• A password file for the Ansible Vault that holds the inventory secrets (see Secrets and hand inputs).
• Network reachability from the controller host to the source cluster management API and to every node managed by it.
• For Classic and Direct paths, sufficient disk space on each source node to cache distribution packages used for rollback.

From the controller to	Purpose
Cloudera Manager API (port 7183 with TLS, or 7180)	Discovery of topology and configuration.
Every cluster node over SSH	Capture of deployed configuration, staging, and the cutover.
Ranger Admin REST API	Capture of the Ranger policy baseline.
Ranger KMS REST API	Capture of encryption-zone key versions (or through a cluster host with --kms-fetch-on).
A Python package index and Ansible Galaxy	Online installation only; the offline variant needs neither.

Identity prerequisites (Modernize and Direct only)

Bifrost deploys Keycloak as part of bifrost modernize land and uses it as the central OIDC Identity Provider for Trino, Airflow, Superset, OpenMetadata, and the Polaris catalog. Two options are supported:

• Bifrost-managed Keycloak (default). The land step installs Keycloak alongside the other platform components, creates the required realms and clients, and wires every service to it automatically. Customers add their users through the Keycloak admin console or via an import job.
• Customer-managed IdP via Keycloak federation. If the customer already runs an enterprise IdP (Okta, Azure AD, Google Workspace, Ping), the shipped Keycloak is configured as an identity broker: it federates to the customer IdP for user authentication while keeping the service-to-service OAuth2 clients local. Bifrost exposes a wrapper values path (keycloak.federation.* in the bifrost modernize land deployment configuration) that is compiled into a realm-import JSON at land time: realm import is the upstream Keycloak mechanism, driven through the keycloakConfigCli component of the underlying Helm chart. Customers can alternatively supply their own realm-import JSON directly via keycloak.keycloakConfigCli.configuration.

Running Bifrost without any Keycloak is not supported: service-to-service OAuth2 between Trino, Polaris, Airflow, and Spark on Ilum depends on it.

DNS and ingress prerequisites (Modernize and Direct only)

The target platform exposes services under a customer-specific domain (the global.domain Helm value). Before bifrost modernize land:

• Provision DNS A or CNAME records pointing <service>.lakehouse.<your-domain> at the ingress controller (or wildcard *.lakehouse.<your-domain>).
• Supply a wildcard TLS certificate or enable cert-manager for per-host certificates. Let's Encrypt works for internet-reachable deployments; for air-gapped deployments, configure cert-manager with an internal CA issuer.
• Open firewall rules from end-user networks to the Keycloak hostname: every browser-based login reaches Keycloak directly.

Target Kubernetes (Modernize and Direct only)

• kubectl 1.28 or later, configured with administrative access to the target cluster.
• helm 3.14 or later.
• A reachable container registry for Spark, Trino, Airflow, and other platform images.
• A Kubernetes cluster with sufficient capacity for the target Ilum deployment. The recommended minimum for a dev or staging deployment is 12 worker nodes, 48 vCPUs, and 192 GB of RAM. Production sizing is described in Operations: capacity planning.

Obtaining Bifrost

Bifrost is delivered through the Ilum Enterprise distribution channel. Customers receive the Bifrost archive and accompanying credentials when the Ilum Enterprise subscription is activated.

Ilum delivers Bifrost as a checksummed archive in one of three forms:

Form	Contents
Assessment bundle (discovery)	Read-only commands for assessing a source estate: discovery, extraction, plan validation, the pre-flight checklist, and the assessment scans. No destructive phase is included.
Classic bundle (classic-full)	The complete Classic toolkit, including the cutover, the management plane takeover, rollback, finalize, the canary, the day-2 commands, the estate runner, and the operator console. It is built for the named clusters of an engagement and drives the clusters it was built for.
Offline variant	Either bundle with every Python package and Ansible collection included, for controllers without network access.

To request access, contact your Ilum account representative or email [email protected].

note

Bifrost is not available through the public Ilum Helm charts. It is not an open-source component and is distributed only to customers with an active Ilum Enterprise agreement.

Installation

Verify the delivered archive, unpack it onto the controller host, and run the installer. The installer verifies every file against the bundle's checksum list, creates a local virtual environment, and installs the Python packages and Ansible collections into it.

# Verify and unpack the Bifrost bundle
sha256sum -c bifrost-<version>-<form>.tar.gz.sha256
tar -xzf bifrost-<version>-<form>.tar.gz
cd bifrost-<version>-<form>

# Install the dependencies into the bundle directory
./install.sh

# Verify the CLI
./bifrost.sh --version
./bifrost.sh classic --help

Commands are run through ./bifrost.sh in the bundle directory, which sets the Ansible configuration and the Python path, or as bifrost after activating .venv. An online installation uses the configured Python package index and Ansible Galaxy; the offline variant needs no network access. No external network access is required during migration itself.

For Modernize and Direct paths, verify that the controller host can reach the target Kubernetes cluster before proceeding.

kubectl cluster-info
helm version

Air-gapped deployments

For Modernize and Direct on air-gapped networks, three mirrors must be prepared in advance:

• Container registry mirror: pull and push every Spark, Trino, Airflow, Polaris, OpenMetadata, Keycloak, Superset, and Ilum image used by the release. The full image list is printed by bifrost modernize land --print-images so customers can script the mirroring pass.
• Helm chart mirror: publish the Bifrost-bundled Helm charts to an internal chart repository (for example, Harbor or ChartMuseum). Point helm repo add at the mirror in the deployment configuration.
• OIDC offline configuration: Keycloak's realm export is prepared on the controller host and loaded during land, so no outbound reachability is required.

Classic paths already ship an on-node distribution package cache and do not need network access during the critical migration window.

Air-gapped controller

For a controller without network access, Ilum delivers the offline variant of the bundle. ./install.sh detects the included packages and installs without contacting any package index. The Ilum Bigtop distribution and the Ambari packages are delivered as local bundles as well and are staged onto the cluster nodes with stage-bigtop and stage-ambari.

Operator Console

Bifrost includes Heimdall, a terminal console for running a migration from the controller. It shows the phase chain of each cluster with its state, verdict, and timing, follows the log of the running phase, and shows the command line behind every action. Start it with ./heimdall (or ./bifrost.sh console) in the bundle directory. See Operator console for a step-by-step guide.

Configuration-as-Data

Bifrost follows a configuration-as-data principle. The migration logic is generic; cluster-specific settings live in YAML inventory files under inventories/. A single Bifrost installation drives migrations for any number of clusters. The only thing that changes between clusters is the inventory. A full Classic toolkit drives the clusters it was built for.

A Classic or Direct inventory describes the source cluster topology. For Classic, hosts.yml is generated by discover and is not edited by hand:

# inventories/prod01/hosts.yml
all:
  vars:
    cluster_name: PROD01
    nameservice: prod01-ns
    zk_quorum: "zk1.example.internal:2181,zk2.example.internal:2181,zk3.example.internal:2181"
    kerberos_realm: EXAMPLE.INTERNAL

  children:
    namenodes:
      hosts:
        nn-01.example.internal: { nn_id: nn1, active: true }
        nn-02.example.internal: { nn_id: nn2, active: false }
    datanodes:
      hosts:
        dn-001.example.internal: { rack: /rack-01 }
        dn-002.example.internal: { rack: /rack-01 }
        # ... one entry per DataNode
    journalnodes:
      hosts:
        jn-01.example.internal: {}
        jn-02.example.internal: {}
        jn-03.example.internal: {}
    hive:
      hosts:
        hive-01.example.internal: {}
    kafka_brokers:
      hosts:
        kafka-01.example.internal: { broker_id: 1 }
        kafka-02.example.internal: { broker_id: 2 }
        kafka-03.example.internal: { broker_id: 3 }

Secrets and hand inputs

Discovery derives most inventory values from the source cluster. Before the first discovery, the operator supplies only what discovery cannot read:

• SSH connection details, in inventories/<cluster>/host_vars/<host>.yml or in inventories/<cluster>/group_vars/all/zz-local.yml.
• The vault password file, inventories/<cluster>/.vault_pass (mode 0600), or the file named by BIFROST_VAULT_PASS_FILE. It is never committed to version control.
• Secrets that the source does not hold, as vault_* values in inventories/<cluster>/group_vars/all/vault.yml, encrypted with Ansible Vault. The Cloudera Manager API password is passed through CM_PASSWORD, or stored with discover --seed-cm-credential.

group_vars/all/ is a directory. main.yml in it is written by discovery, and every derived line is marked as seeded. A value typed by hand permanently outranks a derived value, so values that discovery derives are not typed in. Local overrides belong in zz-*.yml files, which are read after main.yml. Discovery records values that disagree with a hand-typed line in discovery/seed_conflicts.json, and plan validation reports them.

A Modernize or Direct target configuration describes the Kubernetes platform that Bifrost will land:

# modernize/values-production.yaml
global:
  storageClass: rook-ceph-block
  domain: lakehouse.example.internal
  oidc:
    issuerUrl: https://keycloak.example.internal/realms/lakehouse
    clientId: bifrost

rookCeph:
  enabled: true
  cluster:
    objectStore:
      name: lakehouse-rgw
      pool:
        erasureCoded:
          dataChunks: 4
          codingChunks: 2

polaris:
  enabled: true
  replicaCount: 2

trino:
  interactive:
    replicas: 3
    jvm:
      maxHeapSize: "16G"
    config:
      retryPolicy: NONE
  etl:
    replicas: 2
    jvm:
      maxHeapSize: "32G"
    config:
      retryPolicy: TASK

airflow:
  executor: KubernetesExecutor

openmetadata:
  enabled: true
superset:
  enabled: true
keycloak:
  enabled: true

Inventory and values files should be version-controlled, except the vault files, the captured secrets, and .vault_pass, which stay out of version control. Every Bifrost run reads the current inventory state and records its verdicts in the state directory.

Your First Discovery Run

Discovery is non-destructive. It reads the source cluster and produces an inventory, a topology snapshot, and (for Modernize and Direct paths) a scored migration plan. Discovery is safe to run on live production systems.

Classic or Direct discovery against Cloudera CDP

Discovery runs in two passes. The first pass reads the Cloudera Manager API only. The second pass also connects to the cluster nodes over SSH and captures the deployed configuration, the security material, and the encryption zones. Use one lowercase cluster name for every command, and leave --output unset so that later commands find the inventory.

umask 022
export CM_PASSWORD='...'

# Pass 1: Cloudera Manager API only
bifrost classic discover \
  --cluster prod01 \
  --cm-host cm.example.internal \
  --cm-port 7183 \
  --cm-user bifrost-readonly \
  --cm-ca-bundle /etc/pki/bifrost/cm-ca.pem \
  --no-capture-deployed

# Pass 2: full capture over SSH
bifrost classic discover \
  --cluster prod01 \
  --cm-host cm.example.internal \
  --cm-user bifrost-readonly \
  --cm-ca-bundle /etc/pki/bifrost/cm-ca.pem

When one Cloudera Manager manages several clusters, --cm-cluster names the cluster to read. On a Kerberized source with encryption zones, the identities for listing zones and reading key versions are passed with --ez-kinit-keytab, --ez-kinit-principal, --kms-fetch-on, --kms-kinit-keytab, and --kms-kinit-principal. Group options such as --verbose are placed before the command (bifrost classic --verbose discover ...).

Discovery returns PROCEED, or WARN when a part of the capture is incomplete, for example the encryption-zone key proof. It takes tens of minutes on a small cluster and longer on large ones; it only reads from the cluster.

Discovery produces a structured inventory tree:

inventories/prod01/
  hosts.yml                  # Topology inventory
  group_vars/
    all/
      main.yml               # Cluster-wide variables seeded by discovery
      vault.yml              # Operator secrets (Ansible Vault)
    namenodes.yml
    datanodes.yml
    hbase.yml
    edge_nodes.yml
  discovery/
    topology.json            # Hosts, roles, rack assignments
    services.json            # Services, versions, health
    configs/                 # Per-service configuration export
    security.json            # Kerberos settings, keytab locations, TLS certs
    principals.json          # Kerberos principals
    encryption_zones.json    # HDFS encryption zones and key names
    data_dirs.json           # Per-host data directory paths and usage
    client_deps.json         # JDBC driver classes and classpath references
    version_report.json      # Source vs. target version alignment
    required_inputs.json     # Status of every input the migration needs
    secret_capture_probe.json  # Readability of credentials captured from the source
    seed_conflicts.json      # Derived values that disagree with hand-typed lines
    deployed/                # Deployed configuration captured over SSH
    ranger_policy_baseline.json  # Ranger policies before the migration
    service_databases.json   # Service database coordinates (names only)

Modernize discovery against an open-source Hadoop estate

bifrost modernize discover --source-cluster PROD01

Modernize discovery inventories tables, workloads, and policies, and produces a scored migration plan:

modernize/plans/prod01/
  estate_inventory.json       # Tables, jobs, services
  migration_plan.yaml         # Scored plan with wave assignments
  wave_assignments.csv        # Per-table wave assignment and complexity
  tco_report.md               # TCO delta projections
  risk_assessment.md          # Per-component risk analysis
  manual_review_items.json    # Items requiring human judgment

Each table and job is scored on three dimensions:

• Criticality: downstream lineage depth (how many other tables and jobs depend on it).
• Complexity: migration difficulty, derived from workload type (MapReduce and Pig score high; Spark scores low).
• Freshness: last-access timestamp. Tables not accessed in more than 12 months are flagged as candidates for retirement rather than migration.

Review the discovery output

Review the following before proceeding to planning and execution.

• migration_plan.yaml: wave assignments and strategies. Structured YAML, one block per wave, with per-table strategy (snapshot / migrate / add_files), priority, and criticality score.
• manual_review_items.json: items that exceed Bifrost's automation ceiling. Each entry carries component, source_object, reason, and a recommended handling track (for example, custom UDF → "manual port to Trino plugin").
• tco_report.md: markdown narrative projecting monthly operating cost for the legacy and target platforms, broken down by compute, storage, licensing, and support. Drawn from the estate inventory and customer-supplied unit costs.
• risk_assessment.md: markdown table of risks per component (for example, "HBase tables with active coprocessors"), each with severity, affected objects, and mitigation recommendations.

For Classic, review the discovery output in this order:

1. discovery/required_inputs.json: every input the migration needs, with its status (derived, recorded, operator_supplied, not_applicable, or MISSING). A missing input that validate-pre gates on stops the window.
2. discovery/secret_capture_probe.json: for each credential and host, whether the value can be captured from the source. Values themselves are never recorded.
3. discovery/deployed/host_coverage.json: the capture landed on every host. On a TLS cluster, security.json lists the captured keystores and certificates.
4. discovery/encryption_zones.json: the key versions of every zone are captured.
5. discovery/ranger_policy_baseline.json and discovery/service_databases.json.
6. discovery/seed_conflicts.json: derived values that disagree with a hand-typed line.

Discovery can be re-run as many times as needed without affecting the source cluster. Values seeded by an earlier run are kept; delete a seeded line to have it derived again.

After Discovery

The following read-only steps complete the assessment of a Classic migration. None of them changes the cluster:

• bifrost classic extract --cluster prod01 translates the configuration and builds the adoption blueprint. Reports are written to extract/ in the inventory, including the manual-review list in extract/manual_review.json.
• bifrost classic validate-plan --cluster prod01 checks the plan offline and writes extract/plan_validation.md.
• bifrost classic scan-class-coverage --cluster prod01 lists the Java classes the configuration references that the target packages lack.
• bifrost classic scan-metastore-schema compares the source metastore schema with the target schema.
• bifrost classic validate-ambari --cluster prod01 checks that every Cloudera Manager role maps to the target stack.
• bifrost classic validate-pre --cluster prod01 runs the pre-flight checklist and writes its report to the state directory.

See Validation and rollback for the verdict model and Classic migration for the full sequence.

Next Steps

• Choose a path: Classic, Modernize, or Direct.
• Review the CLI reference for every command and flag.
• Understand validation and rollback before executing a migration.
• Prepare the production readiness checklist under Operations.