Limited seats are available to join our revamped training program at Data + AI Summit 2026 in San Francisco, with early-bird pricing that gives you 50% off training if you register by April 30. Key highlights Big savings on training and certification...
Lakebase Postgres now supports customer‑managed keys (CMK), so security teams can keep encryption keys in their own cloud KMS (AWS KMS, Azure Key Vault, or Google Cloud KMS) while Databricks runs Lakebase as a managed service. Key highlights Your key...
Earlier this week, Databricks shared a deep dive into Lakebase Postgres database branching, a “Git‑style” way to work with your databases, meaning you can now create instant, isolated Postgres environments in seconds, without copying all the data or ...
Developers want to write code, not provision databases. Learn how to create and deploy apps directly on the Databricks Platform. Join us on May 5-7. In this webinar, code comes first. Watch experts from Databricks and Cursor demonstrate how to get ...
Community Space for Lakebase Insights Are you building with Lakebase? If so, do you have a single source to stay ahead of every technical shift and architectural breakthrough? More importantly, do you know what practitioners are actually saying o...
We are back with instructor-led Databricks Learning Festivals for APAC & EMEA! Both Udemy and LinkedIn recognized Databricks as one of the top fastest growing skills for 2026 — Udemy's The 100 Fastest-Growing Global Skills for 2026 and LinkedIn's T...
June 15–18, 2026 Register Now! Join thousands of data analytics and AI professionals at Data+ AI Summit 2026, the world’s largest conference for data, analytics and AI in San Francisco and virtually. Don’t miss early-bird pricing! Early-bird discoun...
Our Community Champion Program has always been about recognizing individuals who consistently show up to support others, share their expertise, and strengthen the Databricks Community. Alongside our customers, we’re equally proud of the incredible co...
Customers today expect brands to recognize what they care about, whether they’re reacting to a campaign or actively browsing online. With Databricks Solution Accelerators for propensity scoring on sales data and real‑time clickstream propensity in ec...
The Weekly Digest • April 13 – 19 Community Pulse The deepest dives and the most thoughtful insights from the people who make this space home. Spotlight Contributors The members keeping the ecosystem moving this week: @szymon_dybczak @Sumit_...
I’m experimenting with performance testing in Linux-based environments and was wondering what simple workflows people use to benchmark lightweight apps or workloads.Most examples I find focus on large data pipelines, but I’m more interested in how sm...
Hi TeamI have enrolled in the AI/BI for Data Analyst course. Could you please advise how to get the access for LAB to follow the demo instructions in the courseThanks
@Advika I understand that we need to either have ILT or Databricks lab subscription. Could you just provide the access for the one I boxed it so I don't have to take the full year subscription ? If required, I can upgrade to the full subscription at ...
Every data warehouse has the same foundational problem: keeping dimension tables in sync with operational systems. Customer records change. Orders get updated. Accounts get closed. Getting those changes into your warehouse -- correctly, reliably, and...
Thank you for such an informative post on one of my favourite topics—MERGE INTO.I completely agree that AUTO CDC significantly simplifies handling both SCD Type 1 and Type 2 patterns. The side-by-side comparison between a manual MERGE INTO implementa...
We need to import large amount of Jira data into Databricks, and should import only the delta changes. What's the best approach to do so? Using the Fivetran Jira connector or develop our own Python scripts/pipeline code? Thanks.
Hi @greengil good question, I went through this something similar recently, so sharing what I found.My instinct was also to build it in Python, but once I dug in, the "just write a script" path hides a lot of pain:Deletions are invisible. Jira's RES...
Hi team,When using table update or file arrival trigger, what latency I can expect for the trigger. Does Databricks poll the source by some schedule? If yes, whether the poll is free?Thanks
Hi @Ashwin_DSA ,Thanks for the input. This is very helpful. For the last question, we thought about to create another table as staging, which is specifically used as trigger. Any time source has changes, we will update the staging table too. However ...
Hello Community!I am writing to you with my idea about data ingestion job which we have to implement in our project.The data which we have are in CSV file format and depending on the case it differs a little bit. Before uploading we pivoting csv file...
Hi @maikel, You don't have to build a custom solution for this. Databricks now has native components that align very well with what you want. If you want the job to start as soon as new files land in a volume, the recommended approach is to use file-...
Schema evolution (rescue mode) is being triggered for a complex JSON file. However, when a new column is added in the source, it does not appear in the target table automatically for multi-level nested JSON. How can we add newly added nested columns ...
The solution using the option .option("cloudFiles.schemaEvolutionMode", "addNewColumns") is working as expected. With this configuration, when a new column appears in the source, the stream fails with an UnknownFieldException. However, we would like ...
Hi all,I’m trying to set up REST-based communication between my Lakebase and a REST-client.I’m following the documentation “Connecting to Lakebase via REST using a service principal” to obtain a workspace-level token. After that, I use the Lakebase D...
I created a new Lakebase project to retrace all my steps. 0- I reused my service principal on the workspace1- installed databricks authentication extension: CREATE EXTENSION IF NOT EXISTS databricks_auth;2-Added the lakehouse service principal to the...
Community Space for Lakebase Insights Are you building with Lakebase? If so, do you have a single source to stay ahead of every technical shift and architectural breakthrough? More importantly, do you know what practitioners are actually saying o...
Databricks has launched the Public Preview of Lakeflow Designer, a visual, no‑code, AI‑native way for analysts and business users to prepare and analyze data directly on Databricks without leaving the governed environment of Unity Catalog. Key highli...
Hi,What the maximum size to read using dbutils.fs.head()?is there a limit? because AI says 10MB and I couldn't find useful info in documentations, while I tried in the actual one and it was only limited by the driver memory.Thanks in advance.
dbutils.fs.head() itself does not have a documented hard cap like 10 MB. From the official dbutils reference, the signature is: dbutils.fs.head(file: String, max_bytes: int = 65536): String “Returns up to the specified maximum number of bytes in t...
This is a small deficiency, but a fix would be nice to have.For a long time now, the Sample Data previewer in the Unity Catalog explorer has been unable to show tables that contain a certain kind of column. Instead of showing sample rows of the tabl...
Yes, my vector space is commonly of dimension 4000 or 8000.I don't write any dense vectors to table; can't speak to what happens previewing that type.Thanks for taking up the issue!
Databricks Academy offers the free DevOps Essentials for Data Engineering course, designed to help data engineers apply software engineering best practices and DevOps principles on the Databricks Data Intelligence Platform. Instead of going deep into...
Hi,On a managed Delta table I get:SELECT * FROM abc VERSION AS OF 25;Error:DELTA_UNSUPPORTED_TIME_TRAVEL_BEYOND_DELETED_FILE_RETENTION_DURATION Cannot time travel beyond delta.deletedFileRetentionDuration (168 HOURS).Audit logs show VACUUM START/END...
VACUUM will never delete files on the latest version even if Version 10 was not accessed or modified as it represents the current state of the table. VACUUM targets files that are not referenced by the recent version. It identifies files that were re...
We have a dlt pipeline which creates some same table, which are created based on some transformation and those transformation are kept inside a function in a seperate file. and those file were used using import function.we are deploying those changes...
Hi @Muralidharan_A, To your question about whether retry_on_failure does more than a manual refresh, the answer is yes! retry_on_failure (along with pipelines.numUpdateRetryAttempts and pipelines.maxFlowRetryAttempts) performs classified, timed retri...
| User | Count |
|---|---|
| 2257 | |
| 1313 | |
| 1193 | |
| 1187 | |
| 957 |
| User | Count |
|---|---|
| 417 | |
| 286 | |
| 220 | |
| 63 | |
| 49 |
Every data warehouse has the same foundational problem: keeping dimension tables in sync with operational systems. Customer records change. Orders get updated. Accounts get closed. Getting those chang...
Introduction - Unity Catalog Migration Most organizations running Databricks today started with the Hive Metastore (HMS); it was the default, it worked, and there was no reason to change. But as data ...
Summary Turn unstructured product manuals into structured, queryable data using Databricks Agent Bricks AI Functions, with no custom model training or rigid templates required.Build a complete documen...
Every organization has critical information trapped in PDFs and unstructured documents: forms, reports, records, filings. Historically, turning those files into usable data has meant manual data entr...
