The vector index of a regulatory-document search system quietly filled up. The SDK knew why, but it stayed silent for five minutes and then told me it had "timed out." From firefighting in production to sending a fix upstream, the whole story is worth writing down.

ACT 01 Document Upload Was Broken Again

We run a search system for regulatory documents: public regulations from various countries get chunked, embedded, and stored in Databricks Vector Search, so users can query them in natural language. The system had been stable for months, with hundreds of thousands of document chunks in the index and a few thousand writes a day.

Then one afternoon, customer support reported that document upload was broken again. I pulled up the backend logs: the customer had kicked off a bulk import of regulatory filings, and after waiting... and waiting... five minutes later, this appeared:

2026-10-28 14:56:11 | ERROR | knowledge import failed
TimeoutError: Timed out after 0:05:00
  File "databricks/sdk/retries.py", line 87, in retried
    raise last_err

Timed out after 0:05:00 — an exact five minutes, which happens to be the SDK's default retry timeout. Nothing else in the log.

ACT 02 Debugging Hell: The SDK Was Lying

The usual suspects first: is Databricks down? Rate limiting? A network blip? I ruled them out one by one.

Monitoring first: everything looked fine in the Databricks console. Then request rate: our write volume was far below the documented limits. Then the logs again: nothing but that context-free TimeoutError.

I started to suspect the SDK's retry logic was swallowing the real error. So I bypassed the SDK and sent the same request directly with httpx:

# Throwaway diagnostic script (bypasses the SDK)
import httpx

resp = httpx.post(
    f"{databricks_host}/api/2.0/vector-search/indexes/{index_name}/upsert-data",
    headers={"Authorization": f"Bearer {token}"},
    json={"inputs_json": json.dumps(batch)},
    timeout=30.0,
)
print(resp.status_code, resp.json())

It came back in 1.2 seconds. Status 429, and the response body:

{
  "error_code": "RESOURCE_EXHAUSTED",
  "message": "Endpoint capacity has reached capacity limit. You can delete some data or the index to free up space. Or contact support to increase your quota.",
  "details": [{
    "@type": "type.googleapis.com/google.rpc.RequestInfo",
    "request_id": "b91d04f2-7c33-4e21-9a77-03d2c8e54f10"
  }]
}

Ah. Not rate limiting, not the network — the vector index was out of storage. RESOURCE_EXHAUSTED is a permanent error, and the server had been spelling out the cause all along. But the SDK hid that message, retried against a wall for five minutes, and then pretended it had simply timed out.

The SDK's problem: in customizer.py it sets retry_after_secs for every 429 response without ever looking at the error_code in the response body. To the SDK, RESOURCE_EXHAUSTED (index full) and RATE_LIMIT_EXCEEDED (too many requests) are the same error: wait a bit, try again.

Later I added response-body output where we log, and the production log looked like this:

2026-10-28 15:01:20.118 | WARNING | async_upsert_doc_chunks_batch
429 rate-limited (attempt 13/20), sleeping 10s
body={
  "error_code": "RESOURCE_EXHAUSTED",
  "message": "Endpoint capacity has reached capacity limit."
}

Retry #13, sleeping 10 seconds each time, all wasted effort. The request_id is b91d04f2-..., which lets you trace this exact request in the Databricks backend.

300s	13×	4 lines
The SDK's default retry timeout, all spent waiting for an error that would never recover	Retries the SDK had already burned through by the time we found the problem	Lines of code the final fix required

ACT 03 Firefighting: Splitting a Full Vector Index in Two

Production was on fire, so firefighting first...

Current state: all seven of our vector indexes lived on the same endpoint, and its storage was full. Databricks Direct Access indexes can't be expanded in place, so there was exactly one option: sharding. I created a new endpoint vs-endpoint-2, built a second shard for each index on it, routed new writes to shard 2, and at query time searched both shards concurrently and merged by similarity score.

The concurrent search uses asyncio.gather, roughly like this:

# app/llm/clients.py · DatabricksVSClient.async_similarity_search
async def async_similarity_search(self, query_vector, columns,
                                  filters=None, num_results=10, ...):
    """Query both shards concurrently, merge by score descending."""
    if not self.index_name_2:
        return await asyncio.to_thread(
            self._similarity_search_on, self.index_name, ...
        )

    # Fire both shard requests at once; whoever comes back first just waits
    results1, results2 = await asyncio.gather(
        asyncio.to_thread(self._similarity_search_on, self.index_name, ...),
        asyncio.to_thread(self._similarity_search_on, self.index_name_2, ...),
    )

    merged = sorted(results1 + results2,
                    key=lambda r: r.get('score') or 0.0, reverse=True)
    return merged

On the write path, new data goes to shard 2. No SDK here — straight httpx, precisely because of that five-minute retry behavior:

# Write new data to shard 2; fail fast on RESOURCE_EXHAUSTED, no five-minute wait
target_index = self.index_name_2 or self.index_name

async with httpx.AsyncClient(timeout=60.0) as http:
    resp = await http.post(
        f"{host}/api/2.0/vector-search/indexes/{target_index}/upsert-data",
        content=payload, headers=headers,
    )
    if resp.status_code == 429:
        body = resp.json()
        if body.get('error_code') == 'RESOURCE_EXHAUSTED':
            raise RuntimeError(f'Index is at capacity: {body["message"]}')

The indexes were migrated to shards one after another. Once we confirmed the new endpoint had capacity, the write batch size went from 5 to 55, and throughput jumped more than tenfold. The import job that used to "time out after five minutes" now finishes in under thirty seconds.

ACT 04 The Fire Was Out, but the SDK's Mess Was Still There

With the business back up, I stared at that TimeoutError for a long time.

This wasn't our bug — it was the SDK's. RESOURCE_EXHAUSTED is a permanent error, explicitly defined as non-retryable in the Google API Error Model, yet the Databricks SDK treated it as a rate-limit 429 and retried for five minutes. Worse, the entire error message was swallowed, and what finally surfaced was a meaningless TimeoutError. If we hadn't bypassed the SDK and printed the response body ourselves, pinning this down could easily have taken several times longer.

Reading the SDK source turned up the problem, spread across two files. The call chain:

customizer.py · _RetryAfterCustomizer.customize_error()
if response.status_code in (429, 503): ← looks at the status code only
kwargs["retry_after_secs"] = ... ← RESOURCE_EXHAUSTED gets flagged as "retry"
↓
retries.py · the retried() decorator
if err.retry_after_secs is not None: <sleep & retry> ← is_retryable() is bypassed entirely
↓
TimeoutError: Timed out after 0:05:00 ← the original error is gone

There's a subtler issue underneath: retry_after_secs takes precedence over the is_retryable() callback. Even if someone later marks ResourceExhausted as non-retryable in is_retryable(), it would silently do nothing, because retry_after_secs was already set. So this isn't one isolated bug — it's two design decisions accidentally coupled together.

The fix itself is simple, 4 lines: check error_code before setting retry_after_secs.

# databricks/sdk/errors/customizer.py · the fix
@@ -46,6 +46,13 @@

def customize_error(self, response, kwargs):

    if response.status_code in (429, 503):
+
+        # RESOURCE_EXHAUSTED is permanent, not rate limiting.
+
+        # Retrying won't heal it; let the error propagate to the caller.
+
+        if kwargs.get("error_code") == "RESOURCE_EXHAUSTED":
+
+            return

    kwargs["retry_after_secs"] = self._parse_retry_after(response)

ACT 05 Sent the PR Upstream

The fix was 4 lines. I submitted it upstream to databricks/databricks-sdk-py, with the production log and response body attached as evidence. No breaking changes.

CODA Three Things This Taught Me

1. Never take a timeout error at face value. A TimeoutError is almost never a real timeout — it could be rate limiting, a full index, or an auth failure. An SDK's retry machinery is a black box, and when it tells you "it timed out," the first move is to bypass it and ask HTTP directly.

2. Capacity is the most easily ignored dimension of system design. We had monitoring for performance, latency, and accuracy, but nothing watching the vector index's storage usage. We got lucky this time; with slightly worse luck, it would have been a 3 a.m. incident. That monitoring gap is now closed.

3. An upstream bug isn't just someone else's problem. Once our system was fixed, I could have stopped there. But we had spent hours chasing a problem that should never have been this hard, and the next person to hit RESOURCE_EXHAUSTED would step on the same rake. Four lines of fix, two tests, one PR — trivial to just do.

Open source doesn't mean "whoever wrote it fixes it." It means "whoever finds it gets the chance to fix it."