Hi @Daniel, to help us debug this, could you please share:

1. The streams.json file being used for this job.


2. The output of db.getCollection("mylabtable").getIndexes() from your Mongo shell.

Hi @Daniel, we have released olake version 0.7.3 in which this issue has been fixed. You will have to create a new job to check this.
Please try this version and let us know if you encounter any further issues.

oh so you managed to fix without having to share the streams.json ?

can you tell me if this was only related to mongo?

or also other databases like postgresql and the kafka streams?

found the description here: https://github.com/datazip-inc/olake/pull/951

how can i download my streams.json and recreate a new job? I will have to delete all tables from the lakehouse and full-resync+cdc everything.

@Daniel, the problem was related to primary key discovery in MongoDB. Earlier, the logic detected all indexed fields and treated them as primary keys.
olakeid is generated by concatenating and hashing all the primary keys in a stream. If any field changes, the olakeidchanges, which is why deduplication didn’t work.
You can also check the PR: https://github.com/datazip-inc/olake/pull/951

#951 fix: pin primary key strictly to _id to ensure deterministic olake_id deduplication

<p><strong>Description</strong><br><br>This change resolves a critical deduplication issue in downstream Iceberg tables when syncing MongoDB collections.<br>Previously, <code>ProduceSchema</code> iterated over all indexes of a collection and registered every indexed field as a primary key. Consequently, non-unique indexed fields were included in the stream's <code>SourceDefinedPrimaryKey</code> schema.<br>Because <code>GetKeysHash</code> concatenates all available primary key values to compute the <code>_olake_id</code>, any update to a non-unique indexed field caused the document's <code>_olake_id</code> hash to change. This prevented Iceberg equality-deletes from matching previous records, resulting in duplicate entries instead of correct deduplicated upserts.<br>By specification, MongoDB guarantees that <code>_id</code> is:<br>1. <strong>Mandatory</strong> on every document in standard collections. <a href="https://www.mongodb.com/docs/manual/core/document/#the-_id-field">ref</a><br>2. <strong>Globally Unique</strong> within a collection (enforced by an undeletable unique index).<br>Pinning the primary key solely to <code>_id</code> ensures highly stable, performant, and perfectly deterministic record deduplication.<br>Fixes # (issue)<br><br><strong>Type of change</strong><br><br>• Bug fix (non-breaking change which fixes an issue)<br>• New feature (non-breaking change which adds functionality)<br>• Breaking change (fix or feature that would cause existing functionality to not work as expected)<br>• This change requires a documentation update<br><br><strong>How Has This Been Tested?</strong><br><br>• Deduplication is working fine during updates and deletes for the record, non unique indexed records.<br><br><strong>Screenshots or Recordings</strong><br><br>N/A — Backend driver logic update.<br><br><strong>Documentation</strong><br><br>• Documentation Link: [link to README, <a href="http://olake.io/docs">olake.io/docs</a>, or olake-docs]<br>• N/A (bug fix, refactor, or test changes only)<br><br><strong>Related PR's (If Any):</strong></p>

<a href="https://github.com/datazip-inc/olake">datazip-inc/olake</a>

btw the streams.json takes a few minutes to show up 🙂

image.png

but indexed fields are not primary keys 🙂 so you didn't include the _id if i understand properly?

yeah indexed fields are not primary keys so we have pinned _id as the only primary key as its a mandatory field in every mongo collection (starting from mongo version 3.2+)

also how is a delete treated now? does it delete correctly from iceberg (keeping it in time travel) or will i have a row with every value at null that i will need to filter out?

you will get the op type as d, which can be used to filter out

but i don't want to filter out, i want it to be deleted in iceberg as well

if you have a record A in mongo, which is deleted, the streams from Mongo captures the deletion. Therefore i expect olake to delete the record from iceberg table as well.

@Daniel the record will also be deleted from main table, but as it is soft delete so a row also appears with op type d.

i see thank you!

Hello, just tested and record is still there when delete

2 duplicates _id

the olakeid seems to be well generated but the delete was not performed properly

image.png

@VIKASH CHOUDHARY

hi @Daniel,
Have you created a new job with the new version? Also, could you please share the stream configuration and confirm whether you are running in upsert mode or append-only mode?
Please also share a screenshot of the stream configuration from the Edit Stream page.

Also, are you using partitioning in iceberg table?

yes I have create a new job

and updated the source and destination to v0.7.3

is there an api endpoint to get the stream configuration @VIKASH CHOUDHARY? because through the interface it takes more than 5min to pen

@Daniel For the API endpoint, you can copy the cURL directly from the browser’s Network tab for the stream API request.

here you go

image.png

i am sending you the full json in DM

as you can see it took 3min

image.png

hey @Daniel,
Currently, your equality delete files are going into the null partition. Because of that, the deletes are not being applied correctly to your data, since the records being queried belong to a different partition.
This is happening because, in delete operations, the MongoDB oplog only contains the _id field and does not include the other column values. As a result, Olake is unable to determine the correct partition for the deleted record and creates the equality delete file in the null partition.

To fix this, we will be releasing a new patch soon that adds support for capturing the before image of delete operations from MongoDB oplogs. For this to work, you will need to enable the following setting for each collection:
changeStreamPreAndPostImages: { enabled: true }
Once this is enabled, Olake will be able to read all column values from the delete event and create the equality delete file in the correct partition.

so it is an option to enable with DBAs i guess. but let me ask a question so i am sure i understand, what happens if the partition change? if you have an update of this same record where the partition is year of field sampledate. let's say i have a first record with a sampledate=2025 and then an update to fix data that says the sample_date=2024 (it will write in another patition, will this be considered an update with only one record, or since there are 2 different partitions there will be 2 records in the lakehouse?)

@Daniel

there will be 2 records in the lakehouse

is this a limitation of olake or iceberg? it seems a bit strange to me. because usually the sample_date does not change but it could be an error that needs to be updated and i don't see any field where it could not be an error. we are never safe of that. it s bad if we have duplicated records like that. so what could be the solution?

@Daniel yes its an Iceberg limitation.


But there is a way to handle it. If the previous row is deleted you can identify that from deleted flag that OLake creates (as one of the columns).

So while querying we can always filter out deleted rows.

Hey @Daniel, we have released Olake v0.7.4 with support for capturing deletes from Mongo CDC using pre-images.
Please enable: changeStreamPreAndPostImages: { enabled: true } for each collection.
Once enabled, equality deletes will be written to the correct partition and applied correctly.

The file is corrupt. Here's the breakdown:

  - Magic bytes (PAR1) are present at both start and end — so the file wasn't truncated mid-upload.
  - Footer size is 106.8 MB on a 370 MB file — that's ~29% of the file just for metadata, which is wildly abnormal. A healthy parquet
  footer is usually a few KB to a few MB at most.
  - PyArrow's Thrift deserializer hits its size limit trying to parse that footer, hence Exceeded size limit.

  What likely happened: the writer (RudderStack data plane or whatever produced this file) wrote an enormous number of tiny row
  groups, bloating the footer metadata. Or the footer size field itself got bit-flipped/corrupted during write/flush.

  What you can try:

not urgent as i could probably run a full load, but lmk in either case

@Anirudh Anirudh Can you share the corrupted parquet over the private chat, we will try to check and debug the issue and let you know soon.

sent it @Vaibhav Verma

@Anirudh Anirudh We investigated the file. It does not appear to be corrupt as the PAR1 magic bytes are valid and the row group data is intact.

The failure is caused by oversized Parquet footer metadata. Parquet stores column statistics (min/max values) in the footer, and in this file media_url column contains a ~106 MB base64-encoded MP4 value. That value ended up in the column statistics, inflating the footer to ~106 MB and triggering the reader's 100 MB Thrift deserialization limit.

We’re looking into limiting/truncating metadata generation for such columns to avoid oversized footers.

You can raise an issue for this here: https://github.com/datazip-inc/olake/issues. Also, please let us know if this is urgent on your side

or as a workaround you can try increasing thrift deserialization limit of your downstream jobs greater than your metadata size.

Hi @Vaibhav Verma thanks for catching it, this is definitely a mistake on our end, of pushing the whole mp4 to a db row should not be intended, as for the urgency, i dont think this is very urgent as imo it shouldnt have happened in the first place, anyways, thanks again, good day

Hi @Daniel, can you share the full logs of Kafka sync that is running, worker logs and kafka version? Also can you share all the pods which are running (if you are running via Kubernetes) or else provide all the docker containers running (if you are running via Docker). Also if possible do you know if consumer rebalance has taken place during sync, or the messages of a partition to be synced were expired mid sync? (as per kafka's retention policy)

Hi @Daniel, just checking in, are you still facing this issue?

Can you please share a screenshot of the API error response?
You can get this from the browser’s Network tab:

1. Right click on the page and click "Inspect"


2. Open the "Network" tab


3. Refresh the page and reproduce the issue


4. Click on the failed API request


5. Take a screenshot of the response and share it here

Thread Reply: None

image.png

The error is coming from nginx reverse proxy. Please increase the clientmaxbody_size in your nginx config to 64mb. If it still fails with same error try to increase the value to match the size of your streams. If the issue still persists please let us know.

after i set it to 100mb, it worked. but i got another issue, i setup the job config through UI

image.png

how can i fix it @Nayan Joshi 🙏 tks u so much

@Anh Nguyen Thach could you please download the logs and share it here. You could use the download button at the right bottom in the job logs page. Thanks

here it is

job-5-logs-2026-05-19-04-50-37.tar.gz

@Anh Nguyen Thach Could you please share the source config from the payload hiding all the sensitive info.

You can get it from the network tab:

1. Go to the source page,


2. Click on the edit option on the desired source


3. You can find the property "config" in the response.

{
    "success": true,
    "message": "source '1' retrieved successfully",
    "data": {
        "id": 1,
        "name": "db",
        "type": "postgres",
        "version": "v0.7.3",
        "config": "{\"database\":\"db\",\"host\":\"local.com\",\"max_threads\":3,\"password\":\"SADASDASD\",\"port\":5432,\"retry_count\":3,\"schemas\":[],\"ssh_config\":{\"type\":\"no_tunnel\"},\"ssl\":{\"mode\":\"disable\"},\"update_method\":{\"initial_wait_time\":120,\"publication\":\"olake_uat_publication\",\"replication_slot\":\"olake_uat_s3_slot\",\"type\":\"CDC\"},\"username\":\"db\"}",
        "created_at": "2026-05-18T12:52:27Z",
        "updated_at": "2026-05-18T12:52:27Z",
        "created_by": "admin",
        "updated_by": "admin",
        "jobs": [
            {
                "name": "tst",
                "id": 5,
                "activate": true,
                "destination_name": "sv-uat-shared-data-lake",
                "destination_type": "parquet",
                "last_run_time": "2026-05-19T06:32:00Z",
                "last_run_state": "Running"
            }
        ]
    }
}

@Anh Nguyen Thach are you sure this is the source associated with the Job in the screenshot provided.

I can see the Job Name in the image is "avepoint_sync" and it's id is 2 but I couldn't find the same in the Associated Jobs in the provided payload. Could you please confirm this.

"jobs": [
            {
                "name": "tst",
                "id": 5,
                "activate": true,
                "destination_name": "sv-uat-shared-data-lake",
                "destination_type": "parquet",
                "last_run_time": "2026-05-19T06:32:00Z",
                "last_run_state": "Running"
            }
        ]

sure, i just delete and recrete the job and changed its name

Thread Reply: None

image.png

Are there any ways to fix this? or i must using olake CLI? @Vishal M

@Anh Nguyen Thach We are looking into, will get back shortly.

@Anh Nguyen Thach can we get on a quick call to debug this issue? Let us know a good time anytime today

yes, im free now

Please join here:
https://meet.google.com/fwo-fugn-cyc

meet.google.com

Meet

Real-time meetings by Google. Using your browser, share your video, desktop, and presentations with teammates and customers.

Original URL: https://meet.google.com/fwo-fugn-cyc

cannot seems to add a catalog at all

looking at the logs of UI:

4)
2026-05-21T15:34:17Z WRN failed to fetch image tags online for %s: %s. Cached fallback unavailable on Kubernetes (no Docker daemon)%!(EXTRA string=olakego/source-oracle, **errors.errorString=docker hub api request failed with status code: 504)
[GIN] 2026/05/21 - 15:34:25 | 200 |   2.61ms |       127.0.0.1 | GET      "/api/v1/project/123/destinations"
2026-05-21T15:34:47Z WRN failed to fetch image tags online for %s: %s. Cached fallback unavailable on Kubernetes (no Docker daemon)%!(EXTRA string=olakego/source-mongodb, **errors.errorString=docker hub api request failed with status code: 504)


2026-05-21T15:35:17Z WRN failed to fetch image tags online for %s: %s. Cached fallback unavailable on Kubernetes (no Docker daemon)%!(EXTRA string=olakego/source-kafka, **errors.errorString=docker hub api request failed with status code: 504)
2026-05-21T15:35:47Z WRN failed to fetch image tags online for %s: %s. Cached fallback unavailable on Kubernetes (no Docker daemon)%!(EXTRA string=olakego/source-s3, **errors.errorString=docker hub api request failed with status code: 504)
2026-05-21T15:36:18Z WRN failed to fetch image tags online for %s: %s. Cached fallback unavailable on Kubernetes (no Docker daemon)%!(EXTRA string=olakego/source-db2, **errors.errorString=docker hub api request failed with status code: 504)
2026-05-21T15:36:48Z WRN failed to fetch image tags online for %s: %s. Cached fallback unavailable on Kubernetes (no Docker daemon)%!(EXTRA string=olakego/source-mssql, **errors.errorString=docker hub api request failed with status code: 504)
2026-05-21T15:36:48Z ERR error in request /api/v1/project/123/destinations/versions: failed to get driver image tags: no tags found for image:
[GIN] 2026/05/21 - 15:36:48 | 500 |     4m0s |       127.0.0.1 | GET      "/api/v1/project/123/destinations/versions?type=iceberg"
[GIN] 2026/05/21 - 15:37:11 | 200 |   2.43ms |       127.0.0.1 | GET      "/api/v1/project/123/destinations"
2026-05-21T15:37:19Z WRN failed to fetch image tags online for %s: %s. Cached fallback unavailable on Kubernetes (no Docker daemon)%!(EXTRA string=olakego/source-mysql, **errors.errorString=docker hub api request failed with status code: 504)

@Nayan Joshi

/api/v1/project/123/destinations/ where this rest call come from? I mean with project 123?

@Antonio Yes, OLake requires access to Docker images to fetch and run the connector images.
You currently have 2 options:

1. Use OLake with internet access so it can pull the required images from Docker Hub.


2. Use OLake in an offline setup:

• OLake UI for Offline environments (AWS)


• OLake UI for Offline environments (Generic)

OLake

Offline Environments (AWS)

OLake UI installation in offline and air-gapped network environments on AWS

Original URL: https://olake.io/docs/install/olake-ui/offline-environments-aws/?utm_source=chatgpt.com

OLake

Offline Environments (Generic)

Procedure for updating OLake Stack components and connector images in generic offline environments

Original URL: https://olake.io/docs/install/olake-ui/offline-environments-generic/?utm_source=chatgpt.com

Yes you can definitely do that. Once verified that its a valid issue.

@Antonio we have replied to the issue raised on GitHub, and asked a couple of questions could you please check: https://github.com/datazip-inc/olake-ui/issues/380#issuecomment-4533155841

i'm on holiday 🙂 will come back and check soon

@Antonio We have released the fix for issue https://github.com/datazip-inc/olake-ui/issues/380#issuecomment-4533155841 Please upgrade olake-ui and olake-fusion to latest version to verify the fix.

@Antonio We will check once if this issue is a genuine problem. Once verified, we will update you the same and you can start working on it. Until then please give us some time to check on the issue.

sure 🙂

@Antonio can you retry one more time once?

Yes sure 🙂

Meanwhile, jumping on a different namespace in same catalog

2026-05-26T10:22:29Z ERR error in request /api/opt/v1/airflow/talkdesk/tables: failed to fetch latest Lite process info: failed to get latest MINOR optimizing process for airflow.talkdesk.user_status: failed to get latest MINOR process for airflow.talkdesk.user_status: failed to send request: Get "http://olake-fusion-rest.olake.svc.cluster.local:1630/api/ams/v1/tables/catalogs/airflow/dbs/talkdesk/tables/user_status/optimizing-processes?apiKey=obfuscated: context canceled

Trying to understand where this issue come from - I need to probably see the logs of Polaris as well since could be something there

Regarding the first error above, this is the full error

2026/05/26 10:24:39 new sql script submit, current thread pool state. [Active: 0, PoolSize: 1]
2026/05/26 10:24:39 terminal session dose not exists. create session first
2026/05/26 10:24:39 create a new terminal session.
2026/05/26 10:24:39 fetch terminal session: 1ccf00f5-95e1-429d-9d10-a80be7dc1cb0-null-null-airflow
setup session, session factory: org.apache.amoro.server.terminal.local.LocalSessionFactory
spark.sql.catalog.airflow.catalog-impl  org.apache.iceberg.rest.RESTCatalog
spark.sql.catalog.airflow.table.self-optimizing.quota  0.1
spark.sql.catalog.airflow.client.region  eu-west-1
spark.sql.catalog.airflow.warehouse  airflow
spark.sql.catalog.airflow.uri  
spark.sql.catalog.airflow.table.self-optimizing.enabled  false
spark.sql.catalog.airflow.cache-enabled  false
spark.sql.catalog.airflow.rest.auth.type  oauth2
spark.sql.catalog.airflow.table.self-optimizing.group  spark-container
spark.sql.catalog.airflow.credential  spark.sql.catalog.airflow.table-formats  ICEBERG
spark.sql.catalog.airflow.scope  PRINCIPAL_ROLE:ALL
spark.sql.catalog.airflow.created-at  22 May 2026
spark.sql.catalog.airflow  org.apache.iceberg.spark.SparkCatalog
spark.sql.mixed-format.refresh-catalog-before-usage  true
2026/05/26 10:24:39 session configuration: catalog.airflow.table.self-optimizing.group => spark-container
2026/05/26 10:24:39 session configuration: spark.sql.catalog.airflow.created-at => 22 May 2026
2026/05/26 10:24:39 session configuration: spark.sql.mixed-format.refresh-catalog-before-usage => true
2026/05/26 10:24:39 session configuration: catalog.airflow.uri => https://VW79156.eu-west-1.snowflakecomputing.com/polaris/api/catalog
2026/05/26 10:24:39 session configuration: spark.sql.catalog.airflow.credential => MkvFvBEktCM7s50ZntdvKRxnCS0=:5iJ9qx58XxV7K97OkxdGPah3RRgSm9LY8IY6Zt0gmPY=
2026/05/26 10:24:39 session configuration: session.catalogs => airflow
2026/05/26 10:24:39 session configuration: spark.sql.catalog.airflow.uri => https:///polaris/api/catalog
2026/05/26 10:24:39 session configuration: session.fetch-size => 1000
2026/05/26 10:24:39 session configuration: spark.sql.catalog.airflow.table.self-optimizing.group => spark-container
2026/05/26 10:24:39 session configuration: spark.sql.catalog.airflow.warehouse => airflow
2026/05/26 10:24:39 session configuration: catalog.airflow.warehouse => airflow
2026/05/26 10:24:39 session configuration: spark.sql.catalog.airflow.table.self-optimizing.quota => 0.1
2026/05/26 10:24:39 session configuration: catalog.airflow.cache-enabled => false
2026/05/26 10:24:39 session configuration: catalog-url-base => thrift://127.0.0.1:1260
2026/05/26 10:24:39 session configuration: catalog.airflow.table.self-optimizing.enabled => false
2026/05/26 10:24:39 session configuration: spark.sql.catalog.airflow.catalog-impl => org.apache.iceberg.rest.RESTCatalog
2026/05/26 10:24:39 session configuration: spark.sql.catalog.airflow.client.region => eu-west-1
2026/05/26 10:24:39 session configuration: catalog.airflow.client.region => eu-west-1
2026/05/26 10:24:39 session configuration: session.catalog.airflow.connector => iceberg
2026/05/26 10:24:39 session configuration: spark.sql.catalog.airflow.table-formats => ICEBERG
2026/05/26 10:24:39 session configuration: catalog.airflow.table-formats => ICEBERG
2026/05/26 10:24:39 session configuration: catalog.airflow.table.self-optimizing.quota => 0.1
2026/05/26 10:24:39 session configuration: catalog.airflow.scope => PRINCIPAL_ROLE:ALL
2026/05/26 10:24:39 session configuration: terminal.sensitive-conf-keys =>
2026/05/26 10:24:39 session configuration: catalog.airflow.created-at => 22 May 2026
2026/05/26 10:24:39 session configuration: catalog.airflow.credential => 
2026/05/26 10:24:39 session configuration: catalog.airflow.rest.auth.type => oauth2
2026/05/26 10:24:39 session configuration: spark.sql.catalog.airflow.table.self-optimizing.enabled => false
2026/05/26 10:24:39 session configuration: spark.sql.catalog.airflow.scope => PRINCIPAL_ROLE:ALL
2026/05/26 10:24:39 session configuration: catalog.airflow.catalog-impl => org.apache.iceberg.rest.RESTCatalog
2026/05/26 10:24:39 session configuration: spark.sql.catalog.airflow => org.apache.iceberg.spark.SparkCatalog
2026/05/26 10:24:39 session configuration: spark.sql.catalog.airflow.cache-enabled => false
2026/05/26 10:24:39 session configuration: spark.sql.catalog.airflow.rest.auth.type => oauth2
2026/05/26 10:24:39
2026/05/26 10:24:39 prepare execute statement, line:1
2026/05/26 10:24:39 ALTER TABLE test_olake.test_olake SET TBLPROPERTIES ('self-optimizing.minor.trigger.cron' = '0 ** ** ** **', 'self-optimizing.major.trigger.cron' = '0 **/8 ** ** **', 'self-optimizing.full.trigger.cron' = '', 'self-optimizing.target-size' = '536870912')

@Antonio are you able to connect it?

Sorry After few retries it indeed was able to apply the changes to Amoro 👍

@Ankit Sharma if I need to check the logs for Fusion to run the scheduled maintance, which pods should I look at? of which service?

oh ok yes in the amoro-optimizer saw it

Thread Reply: None

worker.log

@Anh Nguyen Thach can you share sync logs as well, from download logs button from ui

Thread Reply: None

job-17-logs-2026-05-26-10-15-10.tar.gz

@VIKASH CHOUDHARY here it is

hey @Anh Nguyen Thach, Please let us know if you’re available for a call to debug this issue further, either now or at a time that works for you today.

@Anh Nguyen Thach meanwhile, could you please re-run the sync and share the pod logs so we can debug the crash further?
Once the sync pod starts, run:

kubectl logs -f  -n  2>&1 | tee sync.log

also What "context canceled" actually means here?

image.png

Thread Reply: None

image.png

@Anh Nguyen Thach please share complete sync logs

u mean this log? i can't find any error logs, but the pod is still crashing

Logs-2026-06-08 13_34_09.txt

@Anh Nguyen Thach please share from download logs button from ui

Thread Reply: None

job-17-logs-2026-06-08-06-15-40.tar.gz

@Anh Nguyen Thach There are 572 selected streams, and 32 GB RAM is likely too low for a sync of this size. Let's try running it with 128 or 256 GB RAM and see if the issue persists.

What if I create an AWS Glue job for the full load and then trigger an OLake job mode CDC? Will it delete all the data I just loaded @VIKASH CHOUDHARY

@Anh Nguyen Thach What is the max_threads value configured in the source settings? You can check it by clicking Edit Source and reviewing the configuration.

it's 50 as default

@Anh Nguyen Thach You can either reduce the max_threads setting or increase the machine's RAM. Running 50 threads on 32 GB of RAM may not be sufficient for that workload.

Also the same with full refresh. It also break integration.

Hi @Ruslan Zhdan, could you please share the error you are receiving in Snowflake while syncing data using the "Full Refresh" mode in OLake?

Sorry for the delay. I've checked and looks like everything ok regarding to the last case with full refresh.

As far as I can tell it keeps the same table UUID. Correct?

Probably you know during the performing full refresh table will be temporary unavailable?

Hey @Ruslan Zhdan,

1. Yes, the iceberg table-uuid is assigned when the iceberg table is created and is stored in its metadata.json file. It's mostly immutable. From OLake's side, we do not perform any operation that would change the iceberg table-uuid. Happy to know, if there is a specific scenario in your use case where you expect the iceberg table-uuid to change.


2. Regarding table availability during a Full Refresh, could you elaborate a bit more on the scenario which you're referring to?

In case if i have small table which Im going to ingest. Table without any anchor colum with timestamp and records can be updated. Im going to use Full Replace each time and I have some doubts about data consistency and table can be unavailable because Olake perform "Clear destination" first. Is it correct?

And during the write phase. It works with Iceberg branches? Like first of all new snapshot on stage branch and after that merge main?

I've wanted to make sure with full replace table will exists and data will be consistent during writing new snapshot.

I get it, @Ruslan Zhdan


1. Yes. Every time you do a full refresh for the same job, clear destination first drops the iceberg table from the catalog, leading to table unavailability for a short period of time.

Another approach, would be to use append-only mode job for your iceberg table. But, this would create duplicate rows of data for your cdc changes. For now, from the query engine side you can write the logic to dedup the rows.

2. No, towards the write phase, we don't use Iceberg branches right now, we directly create the new snapshot on the main branch.

Internally, we are working on a solution which would make Snowflake, as a query engine, compatible with OLake's Upsert job mode.

Thanks a lot for your support.

@Anh Nguyen Thach We're looking into this internally and will get back to you shortly.

@Anh Nguyen Thach Can u also share datatypes of the null column ?

@Saksham samarth it's jsonB

SELECT
    COUNT(**) AS total_rows,
    COUNT(**) FILTER (WHERE pg_column_size(invoice_details) > 2048) AS rows_over_2kb,
    MAX(pg_column_size(invoice_details)) AS max_size_bytes,
    ROUND(MAX(pg_column_size(invoice_details)) / 1024.0, 2) AS max_size_kb,
    ROUND(AVG(pg_column_size(invoice_details)), 2) AS avg_size_bytes
FROM invoices;

Thread Reply: None

image.png

almost all data in this column is between 3 kb and 15 kb

Hey @Anh Nguyen Thach,
For now, you can use append-only mode as a workaround and write your own deduplication logic on the query side by retaining the required columns.
We'll be picking up support for this soon. In the meantime, you can create a GitHub issue here:
https://github.com/datazip-inc/olake/issues

https://github.com/datazip-inc/olake/issues/984

here it is, thank you so much 🙏

it turned on REPLICA IDENTITY FULL on this table invoices but it still null for that colmn 🥲

image.png

@Anh Nguyen Thach, yes, this is expected. If you're okay with enabling REPLICA IDENTITY FULL and the additional WAL overhead it introduces, we can provide a quick fix for this.

okay, i can enable REPLICA IDENTITY FULL

@Anh Nguyen Thach We're planning to release a fix for this within the next 2 days.

Hey @Anh Nguyen Thach, the fix has been released in v0.7.6. This patch specifically fixes the issue for REPLICA IDENTITY FULL tables.
Please test it when you get a chance and let us know if everything looks good. For previously lost data, you can run Clear Destination and resync.

So I must turn on REPLICA IDENTITY FULL, right? Is there any way to handle this issue without setting it to full in the future?

@Anh Nguyen Thach, yes, for now you'll need to enable REPLICA IDENTITY FULL to handle this case correctly. It's on our roadmap and we do plan to address it, but it's not something we're targeting in the near future. By the way, is the issue resolved on your end after enabling this mode?

is it currently avaiable on helm? im using k8s to host olake

@Anh Nguyen Thach,yes it is available now you can go to edit source and upgrade source version.

seems ui-worker 0.3.8 only support olake v0.7.3 🤔

@Anh Nguyen Thach Can u share the screeshot?

Thread Reply: None

image.png

image.png

image.png

@Anh Nguyen Thach Can u upgrade version in source not in destination.

Hi! You can find the explanation of sync modes in the documentation here: https://olake.io/docs/understanding/terminologies/olake/#2-sync-modes

OLake

OLake Terminologies

Streams Properties Description

Original URL: https://olake.io/docs/understanding/terminologies/olake/#2-sync-modes

@Avinash Kumar 2 At the moment, OLake Fusion provides visibility into compaction runs, logs, and a set of table and run level metrics through the UI. The following documentation covers the currently available monitoring capabilities:

• Runs and Logs


• Metrics

@Nayan Joshi Which version of Olake has this?

@Avinash Kumar 2 You can follow the OLake Fusion setup guide here:
https://olake.io/docs/fusion/getting-started/configure-first-compaction/

The guide covers the prerequisites required to get started with Fusion, whether you're a first-time user or upgrading an existing OLake deployment. It also walks through setting up your first compaction pipeline.

Please let us know if you need any assistance with the setup.

OLake

Configure Your First Compaction

This guide walks through configuring your first compaction for a table.

Original URL: https://olake.io/docs/fusion/getting-started/configure-first-compaction/

@Avinash Kumar 2 If you are using helm and you are a new user please checkout quick start with helm

OLake

Deploy OLake Maintenance on Kubernetes

Deploy OLake Maintenance on Kubernetes with Helm. Enable Maintenance services, configure Spark optimizer (compactor) settings, and run compaction workflows for Iceberg tables.

Original URL: https://olake.io/docs/fusion/install/kubernetes-compaction/#quick-start

@Schitiz Sharma cc.

Hi @Shiv Kumar, we have fixed this issue in helm chart version 0.0.22. Instead of upgrading to version 0.0.21, you can directly upgrade to 0.0.22 and that will fix it.

Ohh my bad, it's working with latest version, thank you.

Hey @Avinash Kumar 2,

For a GCS-backed Lakekeeper using HMAC keys, you can create the warehouse in Lakekeeper with the Flavor set to s3-compat, and then try connecting it through Fusion.

If you still run into any issues, please share your Lakekeeper warehouse configuration and your Fusion Catalog configuration. We'll be more than happy to take a look and help you resolve this.

Do let us know how it goes.

Hey @Badal, What is the recommended or optimal frequency for running CDC jobs in production? For example, is running them every minute generally recommended, or is a 5–15 minute interval a better balance between data freshness and system overhead?

Hey @Avinash Kumar 2,

There is no fix frequency to set for CDC. It primarily depends on the following factors:

• Freshness required in your destination table.


• Change logs accumulation (if not your read disk might start filling up).

@Badal I am unable to select Full Refresh + CDC option. Can you help?

image.png

Hey @Avinash Kumar 2, please make sure you have selected Update Method to CDC, while creating the source.

image.png

@Badal I have have selected Update Method to CDC, while creating the source. Please check

olake_cdc.webm

Hey @Avinash Kumar 2, can you please go through our doc for Setting up MySQL as a Source, and make sure you have all the CDC Prerequisite enabled.

This will help us verify that everything is configured correctly before we proceed.

OLake

OLake MySQL Connector | Setup, Sync Modes &amp; Troubleshooting

Configure OLake MySQL connector for full refresh, CDC; learn prerequisites, connection setup, data type mapping, and common troubleshooting tips

Original URL: https://olake.io/docs/connectors/mysql/

@Badal
All the required CDC prerequisites appear to be enabled, except binlogrowmetadata, which is set to MINIMAL instead of FULL.
We've previously used CDC with Debezium using the same MySQL configuration, and it worked without any issues.
The current configuration is:

SHOW VARIABLES WHERE Variable_name IN (
  'log_bin',
  'binlog_format',
  'binlog_row_image',
  'binlog_row_metadata',
  'gtid_mode'
);
+----------------------+---------+
| Variable_name        | Value   |
+----------------------+---------+
| binlog_format        | ROW     |
| binlog_row_image     | FULL    |
| binlog_row_metadata  | MINIMAL |
| gtid_mode            | ON      |
| log_bin              | ON      |
+----------------------+---------+

cc: @VIKASH CHOUDHARY

hey @Avinash Kumar 2 ,
OLake CDC requires binlogrowmetadata = FULL . OLake relies on binary log events being self-describing to column metadata directly from row events, which requires binlogrowmetadata = FULL.
for more info:
dev.mysql.com/doc/refman/…/replication-options-binary-log.html#…

hey @Jeevansh Gagroo
OLake Go uses the standard MD5 algorithm.
- In the codebase, this is computed via md5.Sum([]byte(concatenatedString)).
for more info:
github.com/datazip-inc/olake/blob/…/utils.go#…

great. thanks.
can you please link this to the documentation here as well for ease of access ?

OLake

Key Data Engineering Terminologies Explained- A Glossary

Understand key terms like data lake, lakehouse, CDC, schema evolution, and concurrency in this OLake glossary for data engineering and replication.

Original URL: https://olake.io/docs/understanding/terminologies/general/?terminology-type=olake#3-record-identification-_olake_id

cc @Merlyn

CC : @Ankit Sharma @Badal

Hey @Avinash Kumar 2, are you using Helm for OLake-Fusion deployment?

In case yes, can you please share a screenshot of the pods running in your Olake namespace? (I just want to confirm whether a pod with optimizer in its name is running.)

Hi @Badal,
Yes optimizer pod is running

2026-06-30 09:47:24.968Z WARN  [main[] [ThriftClientPool] [P:|T:] - Transport open failed, service address: olake-fusion-fusion-optimizing.olake-fusion.svc.cluster.local:1261
2026-06-30 09:47:24.968Z WARN  [main[] [ThriftClientPool] [P:|T:] - Transport open failed, service address: olake-fusion-fusion-optimizing.olake-fusion.svc.cluster.local:1261
2026-06-30 09:47:24.968Z INFO  [main[] [ThriftClientPool] [P:|T:] - Reconnecting service address: olake-fusion-fusion-optimizing.olake-fusion.svc.cluster.local:1261
2026-06-30 09:47:24.968Z INFO  [main[] [ThriftClientPool] [P:|T:] - Reconnecting service address: olake-fusion-fusion-optimizing.olake-fusion.svc.cluster.local:1261
2026-06-30 09:47:25.992Z WARN  [main[] [ThriftClientPool] [P:|T:] - Reconnected service address: olake-fusion-fusion-optimizing.olake-fusion.svc.cluster.local:1261 failed
org.apache.amoro.shade.thrift.org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
    at org.apache.amoro.shade.thrift.org.apache.thrift.transport.TSocket.open(TSocket.java:231) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.shade.thrift.org.apache.thrift.transport.layered.TFramedTransport.open(TFramedTransport.java:82) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.client.ThriftClientPool$1.create(ThriftClientPool.java:100) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.client.ThriftClientPool$1.create(ThriftClientPool.java:72) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:71) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:566) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:306) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:233) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.client.ThriftClientPool.iface(ThriftClientPool.java:189) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.client.OptimizingClientPools.getClient(OptimizingClientPools.java:38) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.optimizer.common.AbstractOptimizerOperator.callAms(AbstractOptimizerOperator.java:53) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.optimizer.common.OptimizerToucher.checkToken(OptimizerToucher.java:74) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.optimizer.common.OptimizerToucher.start(OptimizerToucher.java:59) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.optimizer.common.Optimizer.startOptimizing(Optimizer.java:65) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.optimizer.spark.SparkOptimizer.main(SparkOptimizer.java:69) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) ~[spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1034) ~[spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit.$anonfun$submit$2(SparkSubmit.scala:174) ~[spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62) ~[spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61) ~[spark-core_2.12-3.5.8.jar:3.5.8]
    at java.security.AccessController.doPrivileged(AccessController.java:712) [?:?]
    at javax.security.auth.Subject.doAs(Subject.java:439) [?:?]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) [hadoop-client-api-3.3.4.jar:?]
    at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61) [spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:174) [spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222) [spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) [spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125) [spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134) [spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) [spark-core_2.12-3.5.8.jar:3.5.8]
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.Net.connect0(Native Method) ~[?:?]
    at sun.nio.ch.Net.connect(Net.java:579) ~[?:?]
    at sun.nio.ch.Net.connect(Net.java:568) ~[?:?]
    at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:593) ~[?:?]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
    at java.net.Socket.connect(Socket.java:633) ~[?:?]
    at org.apache.amoro.shade.thrift.org.apache.thrift.transport.TSocket.open(TSocket.java:226) ~[optimizer-job.jar:0.9-SNAPSHOT]
    ... 33 more
2026-06-30 09:47:25.992Z WARN  [main[] [ThriftClientPool] [P:|T:] - Reconnected service address: olake-fusion-fusion-optimizing.olake-fusion.svc.cluster.local:1261 failed
org.apache.amoro.shade.thrift.org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
    at org.apache.amoro.shade.thrift.org.apache.thrift.transport.TSocket.open(TSocket.java:231) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.shade.thrift.org.apache.thrift.transport.layered.TFramedTransport.open(TFramedTransport.java:82) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.client.ThriftClientPool$1.create(ThriftClientPool.java:100) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.client.ThriftClientPool$1.create(ThriftClientPool.java:72) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:71) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:566) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:306) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:233) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.client.ThriftClientPool.iface(ThriftClientPool.java:189) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.client.OptimizingClientPools.getClient(OptimizingClientPools.java:38) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.optimizer.common.AbstractOptimizerOperator.callAms(AbstractOptimizerOperator.java:53) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.optimizer.common.OptimizerToucher.checkToken(OptimizerToucher.java:74) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.optimizer.common.OptimizerToucher.start(OptimizerToucher.java:59) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.optimizer.common.Optimizer.startOptimizing(Optimizer.java:65) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at org.apache.amoro.optimizer.spark.SparkOptimizer.main(SparkOptimizer.java:69) ~[optimizer-job.jar:0.9-SNAPSHOT]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) ~[spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1034) ~[spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit.$anonfun$submit$2(SparkSubmit.scala:174) ~[spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62) ~[spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61) ~[spark-core_2.12-3.5.8.jar:3.5.8]
    at java.security.AccessController.doPrivileged(AccessController.java:712) [?:?]
    at javax.security.auth.Subject.doAs(Subject.java:439) [?:?]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) [hadoop-client-api-3.3.4.jar:?]
    at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61) [spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:174) [spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222) [spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) [spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125) [spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134) [spark-core_2.12-3.5.8.jar:3.5.8]
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) [spark-core_2.12-3.5.8.jar:3.5.8]
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.Net.connect0(Native Method) ~[?:?]
    at sun.nio.ch.Net.connect(Net.java:579) ~[?:?]
    at sun.nio.ch.Net.connect(Net.java:568) ~[?:?]
    at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:593) ~[?:?]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
    at java.net.Socket.connect(Socket.java:633) ~[?:?]
    at org.apache.amoro.shade.thrift.org.apache.thrift.transport.TSocket.open(TSocket.java:226) ~[optimizer-job.jar:0.9-SNAPSHOT]
    ... 33 more
2026-06-30 09:47:27.994Z INFO  [main[] [ThriftClientPool] [P:|T:] - Reconnecting service address: olake-fusion-fusion-optimizing.olake-fusion.svc.cluster.local:1261
2026-06-30 09:47:27.994Z INFO  [main[] [ThriftClientPool] [P:|T:] - Reconnecting service address: olake-fusion-fusion-optimizing.olake-fusion.svc.cluster.local:1261
2026-06-30 09:47:28.017Z INFO  [main[] [OptimizerToucher] [P:|T:] - Adding register property heart-beat-interval:10000 into optimizer
2026-06-30 09:47:28.017Z INFO  [main[] [OptimizerToucher] [P:|T:] - Adding register property heart-beat-interval:10000 into optimizer
2026-06-30 09:47:28.038Z INFO  [main[] [OptimizerToucher] [P:|T:] - Registered optimizer to ams with token:618d2dfd-505a-49b8-bc4c-521ca690dbe0
2026-06-30 09:47:28.038Z INFO  [main[] [OptimizerToucher] [P:|T:] - Registered optimizer to ams with token:618d2dfd-505a-49b8-bc4c-521ca690dbe0

image.png

Hey @Avinash Kumar 2, thanks for sharing the screenshot.

Could please share the complete logs, that would be great for us to debug. kubectl logs > log.txt

@Badal

log.txt

Hey @Avinash Kumar 2, a few questions and requirements:

1. Thanks for sharing the logs — but these are the optimizer logs, not the Fusion (AMS) pod logs. Would be great if you could share the olake-fusion pod logs.


2. Also, please confirm the timestamps for:

1. Since when the optimizer has been running


2. When the Fusion pod was last restarted

2.
a. optimizer has been running for 3h7m
b. No restarts ever since we increased the resources of Fusion pod (It's been running for 3h7m)

log_fusion_pod.txt

Hey @Avinash Kumar 2, can you please re-configure the cron for your iceberg table and check once.

In case you are applying a custom cron, **/5 ** ** ** ** this would be the format we support for an every 5 minute compaciton.

Hey @Badal, I configured like this

image.png

@Avinash Kumar 2 can we connect for a quick meet today? near 3 pm?

@Ankit Sharma Sure
CC: @Sricharan Sirpa

@Avinash Kumar 2 meet.google.com/qba-jjqa-woa

meet.google.com

Meet

Real-time meetings by Google. Using your browser, share your video, desktop, and presentations with teammates and customers.

Original URL: https://meet.google.com/qba-jjqa-woa

@Avinash Kumar 2 I have checked the issue, the table cron is not running because of abruptly kill of AMS, due to which table stuck in planning phase.
We will release a fix for it soon ETA 1 day.
If you need to do it urgently you can follow below instruction
-> stop amoro service
-> run the status update from 500 to 700 in db in table_runtime table (we did it on call)
-> start ams

thanks

@Ankit Sharma Thanks!

@Avinash Kumar 2 we have released new image, if you have set latest and pod restart it will automatically fetch.

image.png

@Avinash Kumar 2 edit on last message:
currently helm has pull policy ifNotPresent that need to be changed to pull automatically,

• either change the pull policy


• or add image tag v0.1.3

@Ankit Sharma we tried the above steps, its working as of now.
Thank you so much
we upgrade the release is done.

Hi @Ankit Sharma, we have started the append-only Kafka ingestion job again. This time, however, compaction doesn't seem to be running for this table. Additionally, the table is not appearing under the Optimizing tab in Amoro.
Could you please help us understand why the compaction isn't being triggered or if there's anything we should check on the Olake/Fusion/Iceberg side?

image.png

image.png

image.png

image.png

@Avinash Kumar 2 could you please share logs of fusion pod?
and the logs of fusion optimizer pod
thanks

Thread Reply: None

log_optimizer.txt

log_fusion2.txt

@Avinash Kumar 2 it seems table is not enabled for compaction, have you taken screenshot after refresh of page?

Also can you just try disabling it and reenabling back ?

While doing this please check the fusion pod logs if you see any error.

@Ankit Sharma Yes, the table is enabled for compaction. I also tried disabling it and enabling it again, but the compaction run still doesn't seem to be starting. I checked the Fusion pod logs during the process as well and couldn't find any related errors.

log_fusion3.txt

can we have a call at 4? @Avinash Kumar 2

@Ankit Sharma Yes sure!

@Avinash Kumar 2 here is the link : meet.google.com/fhs-ekdb-spo

meet.google.com

Meet

Real-time meetings by Google. Using your browser, share your video, desktop, and presentations with teammates and customers.

Original URL: https://meet.google.com/fhs-ekdb-spo

Joining..

Thread Reply: None

job-1-logs-2026-07-08-10-52-01.tar.gz

@Avinash Kumar 2 this is same error that is being discussed here
The issue is with gcp.
we will release a fix for it.

Avinash Kumar 2 (https://getolake.slack.com/team/U0A6088AMM3)

<p>Hi team,<br>We're running a Full Refresh + CDC sync from MySQL at 1-minute frequency, using Lakekeeper (REST catalog) with GCS storage via HMAC keys, warehouse flavor set to <code>s3-compat</code>.<br>Full Refresh completed successfully, and the first few CDC syncs ran fine too — but some recent CDC jobs have started failing.<br>Attached the logs for reference.<br>Thanks!</p>

Original URL: https://getolake.slack.com/archives/C081MV728N7/p1783514459093289

Thread in Slack Conversation

hey @Prasanna Kumar,
olakeid is simply the stringified value of your table's primary key. OLake only generates an MD5 hash (which looks like a random UUID) if it can't detect a single primary key for the source table, could you confirm if your table has a primary key?
olakeid is used as the default identifier field in OLake, which means it automatically dictates both the equality delete column and the sorted_by order.

the table has one (id) auto inc based
While now I understand about the olakeid, The challenge we face is given this is a fast moving table (rapid c and u operations ) , partitioned based on datemodified as day (yyyy/mm/dd), 99% of the updates happen within the same day. The data to delete file ratio is still high. In terms of numbers when queried the $files table,
0 7000+
1 21000 +
2 23800 +

Since the equality deletes are more , 8 files for every data file - compaction on this table is not helping to bring down the equality writes count.

While exploring on how do we over come this challenge ... One of the hint I guessed is to see if the value of olakeid could be tweaked...

Tech Stack

• Olake Sync job on Upsert Mode - CDC Only


• EMR - Spark jobs for compaciton


• AWS Glue - Catalog

hey @Prasanna Kumar, Can you please explain the problem you're currently facing and what outcome or solution you're expecting?

As explained the context above ,
problem

• higher number of delete files than data files .


• compaction does not solved the problem - it consolidated the data files alone

• how to get rid of the delete files - to reclaim the disk space

@Prasanna Kumar
In upsert mode, writing equality delete files cannot be avoided, whereas in append mode, no equality delete files are written. If you wants to reclaim disk space after compaction and remove orphan files, you'll need to configure your compaction job accordingly.

• compaction does not solved the problem - it consolidated the data files alone

Expected Outcome / solution

Understood.... we are running compaction on 1 day older partition (t-1) i.e only on that partition on a daily basis

@Prasanna Kumar please try Fusion once. we are much efficient in compaction. 50% more faster than normal spark. you can run the light compaction at higher frequency to reduce small files overall. which is quite cheap as well

also, if we can have a 15 min call about your usecase. Happy to discuss on it and help

Sure @shubham we can ... Let us first try with Olake fusion on a a controlled env

We started olake-fusion on our uat env. While adding the catalog to olake fusion, I faced some issues and I have few questions:

• Any reason why we are not allowing numbers in the catalog name?


• The info box says if catalog name left empty, it will default to olake_iceberg, but the box is a required field

Screenshot 2026-07-13 at 3.20.42 PM.png

Screenshot 2026-07-13 at 3.21.07 PM.png

Hey @Sandeep Singh, thanks for pointing it out!

1. We'll add a quick fix to support numbers in the Catalog Name field in OLake Fusion. However, we don't plan to support special characters in catalog names.


2. Thanks as well for reporting the incorrect info box message. We'll fix that in the upcoming update too.

Thank you.

Another doubt I had was regarding the driver version. I see there is a change in github.com/datazip-inc/olake/releases?page=2#… regarding the create one positional delete file per data file ref . If I have cdc changes from a version earlier than 0.7.0, are they not compatible fully with spark fusion?

Hey @Sandeep Singh, there's nothing like that. It's fully compatible.

• Any reason why we are not allowing numbers in the catalog name?


• The info box says if catalog name left empty, it will default to olake_iceberg, but the box is a required field

Thank you

Hey @Sandeep Singh, just checking in—did the OLake-UI v0.4.9 release resolve the issue you mentioned above?

Let us know if you're still facing any issues.

hey @Avinash Kumar 2 ,
Could you please share the UI and worker logs? Also, are you using a Helm deployment or Docker?

Also are you unable to view logs for other jobs as well?

@VIKASH CHOUDHARY

• I'm using Helm Deployment


• Currently only one job is running

worker_logs.txt

ui_logs.txt

@Avinash Kumar 2

olake-workers 2026-07-05T08:40:00Z WRN failed to prepare workflow logger for workflowID=sync-123-2-2026-07-05T08:40:00Z: failed to create work directory: mkdir /data/olake-jobs/fe74286797419eb4aa8f462d75f6d8cd0d7ea59d579701ff8ce69a8224b77a2b: no space left on device

@Avinash Kumar 2 are you not using NFS?

@shubham Yes we have enabled NFS server.

@Avinash Kumar 2

please follow this for prod use cases :

olake.io/docs/install/kubernetes#…

OLake

Deploy OLake on Kubernetes | CronJob Setup &amp; Management Guide

Learn to deploy OLake sync tasks as Kubernetes CronJobs with ConfigMaps, PVC, and manual control for automated, recurring data pipelines.

Original URL: https://olake.io/docs/install/kubernetes/#persistent-storage-configuration

Hi @shubham,
We've now moved to an external Filestore-backed NFS instead of the built-in OLake NFS server.
Quick question — is there any built-in TTL / automatic cleanup for the job metadata, config, and log files that OLake writes to the shared storage volume (/data/olake-jobs, /tmp/olake-config)? We noticed these directories aren't purged after a task completes or fails, and they accumulated until our previous NFS volume filled up. Wanted to confirm whether this is expected behavior and if there's a recommended retention/cleanup mechanism, or if we should build our own pruning job.

CC: @Sricharan Sirpa

There is a log cleanup schedule/TTL in our docs. I am AFK. but you can search there. @Avinash Kumar 2

@Avinash Kumar 2 please checkout following doc:
https://olake.io/docs/install/olake-ui/#log-retention

OLake

OLake UI Installation Guide - Docker Compose Setup &amp; Configuration

Quickly deploy OLake UI with Docker Compose. Manage jobs, configure encryption, customize data directory, and troubleshoot common deployment issues.

Original URL: https://olake.io/docs/install/olake-ui/#log-retention

Hi @Avinash Kumar 2, we are looking into this issue , will get back to you soon.

Hi @Vaibhav Verma, just wanted to check if there's any update on this issue, as it will help us plan the next steps. Thanks!

@Avinash Kumar 2, we have figured out the issue and will be doing a release with the fix soon.
Will notify you once release is done

Hi @Avinash Kumar 2, we have fixed this in olake v0.8.2, you can try that version.

If you still face any issue please let us know

@Vaibhav Verma Sure, thanks a lot!

@Avinash Kumar 2 does the new version worked out for you? is your compaction job stable now?

Hey @Kim Tore Jensen !
We are planning to keep this feature for managed SaaS offering of OLake, which is pretty close to its launch. Happy to chat for its beta access to you, if you are interested.

I see. Perhaps another time then. We are not ready to consider a SaaS offering at this time, but thanks for the info 🙂

hey @vikassingh AWS ECS Fargate is not officially supported and we currently don't have any plan to support it any time soon in Olake but we do support docker and helm so you can try that out.

I copied most of the configuration directly from examples in the documentation, modifying only for our Lakekeeper destination.

@Kim Tore Jensen currently there is no continuous feature in CLI. (if you use OLake-ui, we have temporal which continuously syncs data — configurable via cron).

so you need to schedule your own cron/orchestration which will keep running every 10-15mins.

Also we are planning continuous mode in next 2-3 months. You can discuss with our product team @Merlyn about requirements.

Aha! Thanks for clarifying 😊

10-15 mins because of WAL snapshot timings?

@Kim Tore Jensen no its not related to that. you can as low as 20-30 sec as well. depends on your use case.

@Nitish Gupta Thanks for raising the issue ,we will get back to you in soon regarding this issue.

Full logs of a new run that happened just now

olake.log

@Nitish Gupta Thanks for the full logs can you also confirm which deployment mode you are currently using k8s ,docker or CLI ?

docker

@Nitish Gupta could you please let us know whether this is a high priority blocker for your use case with OLake? This will help us prioritize it appropriately. Thanks

Yes it’s a production table that had been running fine for the last 3-4 months, now I had to stop the sync from Olake leaving us with stale data at the source and we’d have to run slow queries on postgres until this is resolved

@Nitish Gupta Sure, we’ll get back to you as soon as possible. Sorry for the inconvenience.

Hey @Nitish Gupta, the current gRPC request limit in OLake is 1 GB, and 10K rows from the CallLog table are exceeding that limit. As a quick fix, could you please turn off this table or any large columns associated with it for now?

Hey @Nitish Gupta, we've picked up this issue internally and will fix it soon.

Hey @Nitish Gupta, we’ve included a patch for this issue in the latest OLake version, v0.9.1. Could you please try it out and let us know if the issue is resolved for you?

hey @Sandeep Singh, existing tables are untouched Iceberg library version and table format version are different things: 1.10.2 is the library, and OLake still pins format-version=2 at table creation

Existing tables will remain backward compatible

understaood, read more on this and clear now

@Grayson Williams

1. Whats the source version of existing sync and new ones? You can find it by editing source. Also did you make any recent changes?


2. You have your replication_slot in place right? Its not deleted or being reused across multiple sources?

@Grayson Williams can you please share your entire logs ,you can find them on Job Logs & History section ?