Processing delay : do you observe the same behavior?

[NBouchia]

In parallel with our tests on version v1.9.0-beta, we have an instance deployed with version 1.8.1.

The server used is in Europe and has the following characteristics:

• RAM : 64 GB DDR4
• Disk : 2 x 512 GB NVMe SSD (Gen4)
• Connection: 1 GBit/s-Port

We do not add any additional CT logs.

The server is not over-used in terms of RAM, CPU, or I/O.

We observe a more or less significant read delay depending on the CT logs.

Steps :

1. Start certstream-server-go
2. Let it run for 48 hours
3. Connect to the full_url route to get the certLink information and compare the processing progress relative to the tree_size after 48 hours

For example after 72h, on argon2026h1, we have around 20M certificate late. Finally, in a day we are able to read less certificate than the log CT add.
Do you observe the same behavior?

[d-Rickyy-b]

I am hosting my own instance on an Oracle Free Tier (ARM) server with 4 vCPUs and 24 GB RAM. I am still running on v1.7.0. And that instance has been up for close to one year non-stop. I checked the commits between 1.8.1 and 1.7.0 and couldn't find any code changes that directly influence the speed certificates are processed at. Only one thing came to my mind and that's the configurable buffer sizes (see this commit and #53). Did you configure those settings?

My server is barely under load. Certstream is also not limited by the CPU.

I connected with a new client and randomly picked multiple certs and compared their not_before dates in order to get a feeling of how recent these certs were issued.

Google xenon2026h1 2222529674: "not_before": 1772993813 - equals Sun Mar 08 2026 18:16:53 GMT+0000
Digicert sphinx 2026h1 897456487: "not_before": 1772992468 - equals Sun Mar 08 2026 17:54:28 GMT+0000.
Digicert wyvern 2026h1 1016274027: "not_before": 1772731953 - equals Thu Mar 05 2026 17:32:33 GMT+0000
TrustAsia log2026a (240305374): "not_before": 1763452019 - equals Tue Nov 18 2025 07:46:59 GMT+0000

Most of the certs were pretty recent. But the IDs were all pretty close to the tree_size. Except for the TrustAsia log. There was a gap of about 743 million certs. From my experience, this sadly highly depends on the CT Log provider / the CA. I barely ever had any issues with Google Logs while Sectigo and TrustAsia Logs were unreliable. In short: I don't observe that behavior - at least on v1.7.0.

Can you share if the issue affected all logs the same? Also, did you set up a grafana dashboard for certstream? Can you see a decline in processing rates for certain logs?

Lastly, what comes to my mind: The CT log operator might limit the bandwidth to your IP.

In parallel with our tests on version v1.9.0-beta, we have an instance deployed with version 1.8.1.

If you're running two instances in parallel, the bandwidth might be limited in a way, so that you can't process all the certificates twice?

[NBouchia]

Thanks you for your response.

Yes, the most observed delay are on Trustasia and Argon2026h1. Other logs are not affected.
Our server start on 27/02/2026, on Argon2026h1 i just checked as you with the not_before information, i saw around 7 days late and on Trustasia around 12 days.

I do not configure buffer_size.
I set up a grafana dashboard and don't see a decline.
Thank you for your point of attention on bandwith. I stop the second instance this morning, i will check this week if there's a change.

To check the delay for all logs, i used this code

from datetime import datetime
import json
import time
from websocket import create_connection

URL_FULL_STREAM = "TO_BE_REPLACED"

DUREE_MAX = 10

def listen_full_stream(ws, log_searched, duree_max=10):
    start_time = time.time()
    for event in ws:
        message = json.loads(event)
        cert_link = message.get("data").get("cert_link")
        if log_searched in cert_link:
            if message['message_type'] == "certificate_update":
                print(f"{datetime.now().strftime("%m/%d/%Y, %H:%M:%S")} -- CertLink: {cert_link} -- not_before : {datetime.fromtimestamp(message['data']['leaf_cert']['not_before'])}")
        if time.time() - start_time > duree_max:
            print("Temps d'écoute écoulé !!")
            break

LOG_LIST = ['ct2026-a.trustasia.com/log2026a',
'ct2026-b.trustasia.com/log2026b',
'ct.googleapis.com/logs/us1/argon2026h1',
'tiger2026h1.ct.sectigo.com',
'ct.googleapis.com/logs/eu1/xenon2026h1',
'elephant2026h1.ct.sectigo.com',
'wyvern.ct.digicert.com/2026h1',
'sphinx.ct.digicert.com/2026h1',
'ct.googleapis.com/logs/us1/argon2026h2',
'wyvern.ct.digicert.com/2026h2',
'sphinx.ct.digicert.com/2026h2',
'ct.googleapis.com/logs/eu1/xenon2026h2',
'tiger2026h2.ct.sectigo.com',
'ct.cloudflare.com/logs/nimbus2026',
'elephant2026h2.ct.sectigo.com',
'ct.cloudflare.com/logs/nimbus2027',
'tiger2027h1.ct.sectigo.com',
'elephant2027h1.ct.sectigo.com',
'ct.googleapis.com/logs/us1/argon2027h1',
'ct.googleapis.com/logs/eu1/xenon2027h1',
'wyvern.ct.digicert.com/2027h1',
'sphinx.ct.digicert.com/2027h1',
'sphinx.ct.digicert.com/2027h2',
'wyvern.ct.digicert.com/2027h2',
'elephant2027h2.ct.sectigo.com',
'tiger2027h2.ct.sectigo.com',
]

if __name__ == '__main__':
    for log in LOG_LIST:
        print(f"Log : {log}")
        listen_full_stream(create_connection(URL_FULL_STREAM, timeout=5), log, DUREE_MAX)

[NBouchia]

Hello,

In the ct-watcher.go file, is it possible to make the two parameters BatchSize and ParallelFetch variable? Is it possible to specify them from the configuration file ? Increasing these two parameters allows more certificates to be processed.

[d-Rickyy-b]

Hi there, currently this is not configurable. I am not sure if that would actually solve the issue you are facing. Your server seems to be a lot more juicy than mine and I don't see any processing delays.

Did you configure the general.recovery feature in the config file? Using that feature for the first time without having a ct_index.json file as a index store will definitely cause the logs to be monitored from the very beginning. That means that certstream will have process billions of certificates to reach the most recent one.

[NBouchia]

Hi, i don't configure the general.recovery feature. Each time i start the server, the process start at the current sth. A delay is observed now for argon2026h2 (US). This log is around 18M certificat by day. I think delay are due to latency. With BatchSize=128 and ParallelFetch=4 we process all entries in one day without delay.

[d-Rickyy-b]

I increased the BatchSize in the dev branch. This will land in the main branch sooner or later ✌️. See also 3ec9251