Cross-cloud conversational analytics made simple with open standards

The reality for most organizations is that data lives everywhere. As engineering teams expand and adopt specialized platforms for their workloads, enterprise data naturally disperses across multiple clouds and operational systems. When analytical and AI initiatives require context from across the entire organization,these teams traditionally face a difficult trade-off: build complex extraction pipelines to copy data into a central repository or accept fragmented reporting. Traditional architectures address this fragmentation with scheduled extraction pipelines that copy remote datasets into a central staging warehouse. This approach creates synchronization lag, adds recurring storage costs, and triggers variable network egress fees. When schemas change upstream, pipelines break, and teams spend hours reconciling metrics across out-of-sync dashboards. BigQuery and Google Cloud’s borderless Lakehouse offer a streamlined way to solve this challenge with a foundation based on the open Apache Iceberg REST catalog specification. Rather than moving data to the compute engine, you can bring Google Cloud compute and generative AI directly to your data, querying it in place across Databricks, Snowflake, and AWS Glue. In this post, we walk through how to configure Databricks Unity Catalog for federation into BigQuery (AWS Glue and Snowflake set up in appendix), run a single query joining data across all three external catalogs, and use conversational analytics in BigQuery to explore the consolidated data with natural language. Figure 1: BigQuery federates metadata across external Apache Iceberg REST catalogs through Lakehouse, enabling in-place conversational analytics without data migration. The borderless lakehouse architecture This solution uses Lakehouse runtime catalog to synchronize metadata from other cloud providers, allowing you to query data with BigQuery or external open source engines through the Apache Iceberg REST catalog endpoint, all without migrating the data. Because Databricks Unity Catalog, Snowflake Horizon Catalog, and AWS Glue Data Catalog support the open Apache Iceberg REST catalog specification, you can apply a consistent connection pattern across all three platforms. When you run a federated query, Lakehouse runtime catalog separates metadata synchronization from storage access: Metadata synchronization: Lakehouse runtime catalog periodically syncs schemas, snapshot pointers, and table definitions from the remote catalog based on a refresh schedule you define. Direct storage access: During query runtime, a compatible engine reads Parquet data files directly from remote object storage, such as Amazon S3, using temporary credentials issued by the remote platform. All query processing runs within Google cloud compute such as BigQuery or Managed Service for Apache Spark. Remote Databricks clusters, Snowflake virtual warehouses, and AWS compute instances are not invoked during execution. Authentication follows an explicit three-step deployment pattern: create the federated catalog in Google Cloud, inspect its configuration to retrieve its automatically generated service account, and grant access permissions within the remote platform. You can use AI to generate sample data. The schema details for tables are provided below. Configure Databricks Unity Catalog We begin in Databricks by preparing the catalog and table. In Unity Catalog, federated access requires that your target schema be associated with an external location registered in Databricks: code_block <ListValue: [StructValue([('code', "-- Run inside Databricks SQL workspacernCREATE CATALOG IF NOT EXISTS dbx_retail_db;rnrn-- Declare the managed target location at the schema levelrnCREATE SCHEMA IF NOT EXISTS dbx_retail_db.dbx_core rn MANAGED LOCATION 's3://my-databricks-bucket/retail/data/';rnrn-- Create the Iceberg dimension tablernCREATE TABLE IF NOT EXISTS dbx_retail_db.dbx_core.dim_products (rn product_key INT,rn product_id STRING,rn product_name STRING,rn category STRING,rn subcategory STRING,rn brand STRING,rn price DOUBLE,rn cost DOUBLErn)rnUSING iceberg;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30e66290>)])]> You can use AI to generate sample data. The schema details for tables are provided in each section. Before external engines can read this table, a Databricks metastore administrator must enable External data access on the metastore details tab. Go to Databricks Admin account and under User Management, create a Databricks Service Principal and make a note of its UUID "6ea7b861-c5a5-xxxx-xxxx-xxxxxxxxxxxx" Connect Databricks to BigQuery Next, create the federated catalog inside Google Cloud using the gcloud command-line tool. You will need your Databricks workspace URL and the application ID of your Databricks service principal: code_block <ListValue: [StructValue([('code', 'export GCP_PROJECT_ID="my-lakehouse-project"rnexport GCP_REGION="us-central1"rnrngcloud alpha biglake iceberg catalogs create dbx_retail_db rn --project="GCP_PROJECT_ID" rn --primary-location="GCP_REGION" rn --catalog-type="federated" rn --federated-catalog-type="unity" rn --unity-instance-name="dbc-xxxxxxxx-xxxx.cloud.databricks.com" rn --unity-catalog-name="dbx_retail_db" rn --unity-service-principal-application-id="6ea7b861-c5a5-xxxx-xxxx-xxxxxxxxxxxx" rn --refresh-interval="300s"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e315db8d0>)])]> Once created, describe the catalog to view its assigned service account: code_block <ListValue: [StructValue([('code', 'gcloud alpha biglake iceberg catalogs describe dbx_retail_db rn --project="GCP_PROJECT_ID"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e315da010>)])]> Return to Databricks to assign permissions to your service principal: code_block <ListValue: [StructValue([('code', '-- Grant catalog traversal permissionrnGRANT USE CATALOG ON CATALOG dbx_retail_db rn TO 6ea7b861-c5a5-xxxx-xxxx-xxxxxxxxxxxx;rnrn-- Grant schema and read permissionsrnGRANT USE SCHEMA,EXTERNAL USE SCHEMA, SELECT ON SCHEMA dbx_retail_db.dbx_core rn TO 6ea7b861-c5a5-xxxx-xxxx-xxxxxxxxxxxx;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e317dd090>)])]> You must grant the EXTERNAL USE SCHEMA privilege to the Databricks Service Principal on the target schemas or catalog to enable credential vending for external data access. Finally, to complete the connection, configure the federation policy on the Databricks service principal with the following details - Issuer URL* - https://accounts.google.com Subject* - numeric subject ID returned by the catalog describe command eg 1124251400450xxxxxxxx Audiences - https://accounts.cloud.databricks.com Run one query across three clouds Federated Iceberg tables are addressed in BigQuery using four-part paths: project.catalog.namespace.table. In our environment, sales transactions reside natively in BigQuery: code_block <ListValue: [StructValue([('code', 'CREATE SCHEMA IF NOT EXISTS local_retail_datasetrnOPTIONS(location="us-central1");rnrnCREATE OR REPLACE TABLE my-lakehouse-project.local_retail_dataset.fact_sales (rn sale_id STRING,rn sale_date DATE,rn customer_key INT64,rn product_key INT64,rn store_key INT64,rn quantity INT64,rn unit_price NUMERIC,rn discount NUMERIC,rn total_revenue NUMERIC,rn total_cost NUMERIC,rn margin NUMERICrn)rnPARTITION BY sale_daternCLUSTER BY store_key, product_key;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e317dfa10>)])]> We can now run a single SQL query joining our native sales transactions with customer records in Snowflake, product lines in Databricks, and store locations in AWS Glue: code_block <ListValue: [StructValue([('code', "SELECT rn c.first_name || ' ' || c.last_name AS customer_name, -- From Snowflake Icebergrn p.product_name, -- From Databricks Icebergrn s.store_name, -- From AWS S3 Icebergrn SUM(f.quantity) AS total_units_sold, -- From BigQuery native factrn SUM(f.total_revenue) AS total_sales_revenue -- From BigQuery native factrnFROM my-lakehouse-project.local_retail_dataset.fact_sales frnINNER JOIN my-lakehouse-project.sf_retail_db.CORE.DIM_CUSTOMERS c rn ON f.customer_key = c.customer_keyrnINNER JOIN my-lakehouse-project.dbx_retail_db.dbx_core.dim_products p rn ON f.product_key = p.product_keyrnINNER JOIN my-lakehouse-project.glue_retail_db.glue_retail_db.dim_stores s rn ON f.store_key = s.store_keyrnGROUP BY 1, 2, 3rnORDER BY total_sales_revenue DESCrnLIMIT 10;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e317dc8d0>)])]> BigQuery processes the joins in place. Notice that Snowflake identifiers appear in uppercase (CORE.DIM_CUSTOMERS), while Databricks and AWS Glue maintain lowercase table paths. Figure 2 - Query results combining data across Snowflake, Databricks, and AWS Glue. Explore cross-cloud data with natural language Because federated Iceberg tables behave like native tables within BigQuery, downstream analytical features inherit access directly. You can point conversational analytics in BigQuery at these federated schemas to let team members explore cross-cloud data using natural language prompts. To give conversational analytics full operational context, we save our federated join as a logical view: code_block <ListValue: [StructValue([('code', "CREATE OR REPLACE VIEW my-lakehouse-project.local_retail_dataset.v_omnichannel_sales_pulse ASrnSELECT rn f.sale_id,rn f.sale_date,rn c.first_name || ' ' || c.last_name AS customer_name,rn c.country AS customer_country,rn p.product_name,rn p.category AS product_category,rn p.brand AS product_brand,rn s.store_name,rn s.city AS store_city,rn s.region AS store_region,rn f.quantity,rn f.total_revenue,rn f.marginrnFROM my-lakehouse-project.local_retail_dataset.fact_sales frnINNER JOIN my-lakehouse-project.sf_retail_db.CORE.DIM_CUSTOMERS c rn ON f.customer_key = c.customer_keyrnINNER JOIN my-lakehouse-project.dbx_retail_db.dbx_core.dim_products p rn ON f.product_key = p.product_keyrnINNER JOIN my-lakehouse-project.glue_retail_db.glue_retail_db.dim_stores s rn ON f.store_key = s.store_key;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30dc3090>)])]> Analyze sales performance in natural language When you ask conversational analytics in BigQuery: "Who are our top customers by total sales revenue, what products did they buy, and which stores did they shop at?" Conversational analytics evaluates the schema across all three remote catalogs, generates the SQL statement, executes the query, and presents the response. Figure 3: Conversational analytics in BigQuery translates plain-language prompts into federated queries across Snowflake, Databricks, and AWS Glue. Plan for caching and data residency Before deploying federated catalogs in production, consider these operational behaviors: Background refresh timing: The --refresh-interval flag controls metadata synchronization. The minimum interval is 300 seconds. For larger catalogs containing hundreds of tables, metadata retrieval can take longer. If a sync cycle overruns, BigQuery skips the next scheduled run to avoid task queuing. Automatic query caching: BigQuery caches retrieved cross-cloud data blocks in the region where your federated catalog resides (e.g. us-central1). This helps reduce network transfer on repeated queries. Because cached blocks reside within Google Cloud storage in that region, confirm that this aligns with your organization's data residency policies. Encryption controls: Cross-cloud cache blocks use Google-managed encryption keys. If your organization enforces the constraints/gcp.restrictNonCmekServices policy constraint, BigQuery bypasses the cache and reads directly from remote storage for every query. Monitoring access metrics: You can evaluate cache utilization in your query execution plans. Check the objectStorageStats section in BigQuery job statistics to compare cacheBytesRead against objectStorageBytesRead. Next steps In summary, Google Cloud’s borderless Lakehouse helps you activate your data regardless of where it lives without costly migrations or building additional data pipelines. Get started today by reviewing the borderless Lakehouse architecture documentation to learn more about supported regions and credential configurations. You can work through the borderless Lakehouse codelab to practice federating external catalogs step-by-step. Finally, connect with your Google Cloud account team or customer engineer to evaluate borderless lakehouse patterns for your specific data platforms or industries. Appendix Setting up Snowflake and AWS Glue cross cloud data access Connect Snowflake and Big Query Configure Snowflake Horizon Catalog We begin in Snowflake by creating the database, schema, and Iceberg table. In Snowflake Horizon, managed Iceberg tables expose an open Apache Iceberg REST catalog interface automatically: code_block <ListValue: [StructValue([('code', "-- Run inside Snowflake SQL worksheetrnCREATE DATABASE IF NOT EXISTS sf_retail_db;rnCREATE SCHEMA IF NOT EXISTS sf_retail_db.core;rnrn-- Create the Iceberg dimension tablernCREATE OR REPLACE ICEBERG TABLE sf_retail_db.core.dim_customers (rn customer_key INTEGER,rn customer_id VARCHAR,rn first_name VARCHAR,rn last_name VARCHAR,rn email VARCHAR,rn gender VARCHAR,rn country VARCHAR,rn signup_date DATErnrn)rnCATALOG = 'SNOWFLAKE'rnEXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED';"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30dc1b50>)])]> Before external engines can read this catalog, configure a dedicated reader role and grant traversal permissions across the database, schema, and table. SQL code_block <ListValue: [StructValue([('code', '-- Create a dedicated reader rolernCREATE ROLE IF NOT EXISTS gcp_lakehouse_reader_role;rnrn-- Grant database and schema traversal permissionsrnGRANT USAGE ON DATABASE sf_retail_db TO ROLE gcp_lakehouse_reader_role;rnGRANT USAGE ON SCHEMA sf_retail_db.core TO ROLE gcp_lakehouse_reader_role;rnrn-- Grant read access on the target tablernGRANT SELECT ON TABLE sf_retail_db.core.dim_customers TO ROLE gcp_lakehouse_reader_role;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30dc3490>)])]> Next, create the federated catalog inside Google Cloud using the gcloud command-line tool. You will need your Snowflake account identifier, the reader role, and the root Iceberg database name: bash code_block <ListValue: [StructValue([('code', 'export GCP_PROJECT_ID="my-lakehouse-project"rnexport GCP_REGION="us-central1"rnrngcloud alpha biglake iceberg catalogs create sf_retail_db rn --project="GCP_PROJECT_ID" rn --primary-location="GCP_REGION" rn --catalog-type="federated" rn --federated-catalog-type="snowflake" rn --snowflake-account-identifier="xy12345-ab67890" rn --snowflake-warehouse="SF_RETAIL_DB" rn --snowflake-role="gcp_lakehouse_reader_role" rn --refresh-interval="300s"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e317dd4d0>)])]> In the Apache Iceberg REST specification, --snowflake-warehouse references the catalog root database name (SF_RETAIL_DB), not a virtual compute warehouse. Once created, describe the catalog to view its assigned numeric service account ID: bash code_block <ListValue: [StructValue([('code', 'gcloud alpha biglake iceberg catalogs describe sf_retail_db rn --project="GCP_PROJECT_ID" rn --format=value(biglake-service-account-id)''), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30dc2b90>)])]> Return to Snowflake to bind Google Cloud's numeric identity to a passwordless service user via Workload Identity Federation (WIF): SQL code_block <ListValue: [StructValue([('code', "-- Run inside Snowflake SQL worksheetrnUSE ROLE ACCOUNTADMIN;rnrn-- Bind Google Cloud's numeric identity to a passwordless service userrnCREATE OR REPLACE USER snowflake_service_userrn TYPE = SERVICErn WORKLOAD_IDENTITY = (rn TYPE = GCPrn SUBJECT = '102669xxxxxxxxxxxxxxxxx' -- Numeric subject ID returned by gcloud describern )rn DEFAULT_ROLE = gcp_lakehouse_reader_rolern COMMENT = 'Service user for Lakehouse catalog federation over WIF';rnrn-- Grant the reader role to this service userrnGRANT ROLE gcp_lakehouse_reader_role TO USER snowflake_service_user;"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30dc2950>)])]> This establishes passwordless authentication over OpenID Connect, allowing BigQuery to discover table metadata and vend temporary query-time credentials directly from Snowflake Horizon. Configure AWS Glue Data Catalog We begin in AWS by preparing the IAM identity, establishing access permissions across AWS Glue and Amazon S3, and registering the Iceberg table. First, create a placeholder trust policy named trust_policy.json. This allows you to provision the AWS IAM role before Google Cloud assigns the unique catalog service agent: code_block <ListValue: [StructValue([('code', '{rn "Version": "2012-10-17",rn "Statement": [rn {rn "Effect": "Allow",rn "Principal": {rn "Federated": "accounts.google.com"rn },rn "Action": "sts:AssumeRoleWithWebIdentity",rn "Condition": {rn "StringEquals": {rn "accounts.google.com:aud": ["PLACEHOLDER_VALUE"],rn "accounts.google.com:sub": ["PLACEHOLDER_VALUE"]rn }rn }rn }rn ]rn}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30dc0d10>)])]> Create the IAM role using the AWS CLI. Setting a 12-hour session duration (43200 seconds) prevents credential expiration during long-running analytical queries: code_block <ListValue: [StructValue([('code', 'export AWS_ROLE_NAME="lakehouse_glue_federation_role"rnrnaws iam create-role rn --role-name "AWS_ROLE_NAME" rn --assume-role-policy-document file://trust_policy.json rn --max-session-duration 43200'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30dc3f90>)])]> Next, define the permissions policy in a file named permissions_policy.json. This policy grants BigQuery permission to read the AWS Glue catalog metadata and access the underlying Parquet data files in Amazon S3: code_block <ListValue: [StructValue([('code', '{rn "Version": "2012-10-17",rn "Statement": [rn {rn "Sid": "GlueCatalogRead",rn "Effect": "Allow",rn "Action": [rn "glue:GetCatalog",rn "glue:GetDatabase",rn "glue:GetDatabases",rn "glue:GetTable",rn "glue:GetTables"rn ],rn "Resource": "arn:aws:glue:us-east-2:123456789012:"rn },rn {rn "Sid": "S3BucketAndObjectRead",rn "Effect": "Allow",rn "Action": [rn "s3:ListBucket",rn "s3:GetObject"rn ],rn "Resource": [rn "arn:aws:s3:::my-glue-iceberg-bucket",rn "arn:aws:s3:::my-glue-iceberg-bucket/"rn ]rn }rn ]rn}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30dc0f90>)])]> Attach this permissions policy to your newly created role: code_block <ListValue: [StructValue([('code', 'aws iam put-role-policy rn --role-name "AWS_ROLE_NAME" rn --policy-name "lakehouse_glue_federation_policy" rn --policy-document file://permissions_policy.json'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30dc1850>)])]> Important (AWS Lake Formation): If your AWS environment uses AWS Lake Formation, verify that the IAMAllowedPrincipals group has access to the target database and table (or grant explicit Lake Formation permissions to lakehouse_glue_federation_role if the bucket is registered with AWS Lake Formation). Without this, Lake Formation security settings will block the IAM role from accessing catalog metadata. With permissions in place, create the database and Iceberg table in AWS Glue using Amazon Athena: code_block <ListValue: [StructValue([('code', "-- Run inside Amazon AthenarnCREATE DATABASE IF NOT EXISTS glue_retail_db;rnrnCREATE TABLE IF NOT EXISTS glue_retail_db.dim_stores (rn store_key INT,rn store_id STRING,rn store_name STRING,rn city STRING,rn state STRING,rn region STRING,rn manager STRING,rn opened_date DATErnrn)rnLOCATION 's3://my-glue-iceberg-bucket/retail/dim_stores/'rnTBLPROPERTIES (rn 'table_type'='ICEBERG', rn 'format'='parquet', rn 'write_compression'='zstd' rnrn);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30dc2150>)])]> Connect AWS Glue to BigQuery Next, register the federated catalog inside Google Cloud using the gcloud command-line tool. You will need your 12-digit AWS Account ID, AWS Region, and the role ARN created above: code_block <ListValue: [StructValue([('code', 'export GCP_PROJECT_ID="my-lakehouse-project"rnexport GCP_REGION="us-central1"rnexport AWS_REGION="us-east-2"rnexport AWS_ACCOUNT_ID="123456789012"rnrngcloud alpha biglake iceberg catalogs create glue_retail_db rn --project="GCP_PROJECT_ID" rn --primary-location="GCP_REGION" rn --catalog-type="federated" rn --federated-catalog-type="glue" rn --glue-warehouse="AWS_ACCOUNT_ID" rn --glue-aws-region="AWS_REGION" rn --glue-aws-role-arn="arn:aws:iam::{AWS_ACCOUNT_ID}:role/lakehouse_glue_federation_role"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30dc0490>)])]> To prevent premature background refresh failures while the AWS trust policy propagates, initialize the catalog without setting a refresh schedule. Next, extract the unique numeric subject ID assigned to the catalog by Google Cloud: code_block <ListValue: [StructValue([('code', 'export LAKEHOUSE_SA_ID=(gcloud alpha biglake iceberg catalogs describe glue_retail_db rn --project="GCP_PROJECT_ID" rn --format="value(biglake-service-account-id)")rnrnecho "Assigned Lakehouse Service Account ID: LAKEHOUSE_SA_ID"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30dc2f50>)])]> Return to the AWS CLI and update your IAM role's trust policy, replacing the placeholder with the verified Google Cloud identity: code_block <ListValue: [StructValue([('code', 'cat > trust_policy_final.json << EOFrn{rn "Version": "2012-10-17",rn "Statement": [rn {rn "Effect": "Allow",rn "Principal": {rn "Federated": "accounts.google.com"rn },rn "Action": "sts:AssumeRoleWithWebIdentity",rn "Condition": {rn "StringEquals": {rn "accounts.google.com:aud": [rn "LAKEHOUSE_SA_ID"rn ],rn "accounts.google.com:sub": [rn "LAKEHOUSE_SA_ID"rn ]rn }rn }rn }rn ]rn}rnEOFrnrnaws iam update-assume-role-policy rn --role-name "AWS_ROLE_NAME" rn --policy-document file://trust_policy_final.json'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30dc1050>)])]> 5. Enable metadata synchronization and verify bash code_block <ListValue: [StructValue([('code', 'gcloud alpha biglake iceberg catalogs update glue_retail_db rn --project="GCP_PROJECT_ID" rn --refresh-interval="300s"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30dc3890>)])]> Verify that the namespaces are discovered and synchronized: bash code_block <ListValue: [StructValue([('code', '# Verify namespaces synchronized from AWS Gluerngcloud alpha biglake iceberg namespaces list rn --project="$GCP_PROJECT_ID" rn --catalog="glue_retail_db"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f7e30dc07d0>)])]>

Source: Google Cloud AI | 2026-10-13

Related