Pub/Sub Topic batch auto commit? #2382

Activity

1. daspecster

   added

   type: questionRequest for information or clarification. Not an issue.Request for information or clarification. Not an issue.

   api: pubsubIssues related to the Pub/Sub API.Issues related to the Pub/Sub API.

   on Sep 22, 2016
2. daspecster commented on Sep 22, 2016

   

   daspecster

   on Sep 22, 2016

   Last edited by daspecster

   Contributor

   More actions

   By committing do you mean publishing a message?

       with topic.batch() as batch:
           batch.publish(PAYLOAD1)
           batch.publish(PAYLOAD2, extra=EXTRA)

   From https://googlecloudplatform.github.io/google-cloud-python/stable/pubsub-usage.html#publish-messages-to-a-topic

   @tseaver could something like this be done?

       with topic.batch() as batch:
           for msg in xrange(100):
               if len(batch.messages) > 10:
                    batch.commit()
               batch.publish(msg)  # msg would be 0, 1, 2...
3. dhermes commented on Sep 22, 2016

   

   dhermes

   on Sep 22, 2016

   Contributor

   More actions

   @tcroiset It makes sense, though in this case you probably would rather use it outside of a context manager, e.g.

   batch = topic.batch()
   for msg in messages:
       batch.publish(msg)
   
   batch.commit()

   NOTE: There is no batch.begin() to call, as can be seen by the context manager implementation.

   NOTE: Calling batch.commit() at the end is to do any clean-up if necessary, but if there are no unpublished messages commit() will do nothing.
4. tseaver commented on Sep 23, 2016

   

   tseaver

   on Sep 23, 2016

   Contributor

   More actions

   ISTM that @tcroiset was asking for something like:

   from google.cloud.pubsub.topic import Batch
   
   class MaxMessagesBatch(Batch):
   
       def __init__(self, topic, client, max_messages):
           super(MaxMessagesBatch, self).__init__(topic, client)
           self._max_messages = max_messages
   
       def publish(self, message, **attrs):
           super(MaxMessagesBatch, self).publish(message, **attrs)
           if len(self.messages) >= self._max_messages:
               self.commit()

   👍React with 👍1Reacted by Danny Hermes
5. tcroiset commented on Sep 29, 2016

   

   tcroiset

   on Sep 29, 2016

   Author

   More actions

   Indeed.

   If i'm not mistaken, there's a limit (1000?) on the api side for the number of messages published in a single call.
   I have implemented a class like MaxMessagesBatch but I was thinking it could be directly implemented in gcloud.
6. tseaver commented on Oct 12, 2016

   

   tseaver

   on Oct 12, 2016

   Contributor

   More actions

   @tcroiset FWIW the API docs (JSON | gRPC) don't mention any limit.
7. tseaver

   added

   type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.‘Nice-to-have’ improvement, new feature or different behavior or design.

   and removed

   type: questionRequest for information or clarification. Not an issue.Request for information or clarification. Not an issue.

   on Oct 12, 2016
8. tcroiset commented on Oct 18, 2016

   

   tcroiset

   on Oct 18, 2016

   Author

   More actions

   @tseaver It is mentioned here: https://cloud.google.com/pubsub/quotas

   Publish request
   Total size must be no more than 10MB .
   Must contain no more than 1,000 messages.
9. lukesneeringer

   closed this as completedon Apr 19, 2017
10. JustinBeckwith

    assigned

    lukesneeringer

    on Feb 1, 2021
11. parthea

    added a commit that references this issue on Nov 24, 2025

    chore(main): release 1.24.1 (#2382)

    ...

    2439c99
12. parthea

    added a commit that references this issue on Apr 1, 2026

    refactor: fix some aggregation ops in the sqlglot compiler (#2382)

    ...

    b00226b