New 'MessageToDict' implementation breaks with unknown entry types

>>> from google.cloud.logging import Client
>>> client = Client()
>>> def do_something_with(entry): pass
... 
>>> for entry in client.list_entries():
...     do_something_with(entry)
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/cloud/iterator.py", line 211, in _items_iter
    for item in page:
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/cloud/iterator.py", line 155, in next
    result = self._item_to_value(self._parent, item)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/cloud/logging/_gax.py", line 478, in _item_to_entry
    resource = MessageToDict(entry_pb)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/protobuf/json_format.py", line 133, in MessageToDict
    return printer._MessageToJsonObject(message)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/protobuf/json_format.py", line 164, in _MessageToJsonObject
    return self._RegularMessageToJsonObject(message, js)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/protobuf/json_format.py", line 196, in _RegularMessageToJsonObject
    js[name] = self._FieldToJsonObject(field, value)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/protobuf/json_format.py", line 230, in _FieldToJsonObject
    return self._MessageToJsonObject(value)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/protobuf/json_format.py", line 162, in _MessageToJsonObject
    return methodcaller(_WKTJSONMETHODS[full_name][0], message)(self)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/protobuf/json_format.py", line 266, in _AnyMessageToJsonObject
    sub_message = _CreateMessageFromTypeUrl(type_url)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/protobuf/json_format.py", line 341, in _CreateMessageFromTypeUrl
    'Can not find message descriptor by type_url: {0}.'.format(type_url))
TypeError: Can not find message descriptor by type_url: type.googleapis.com/google.cloud.audit.AuditLog.

@tseaver

@tseaver Yup, @bjwatson had reported this earlier and I confirmed with @waprin yesterday. Sorry for not filing. I'd been meaning to resolve this.

@tseaver

@tseaver @dhermes We're tracking this in https://github.com/googleapis/googleapis/issues/187.

What's the relative priority? We're focused on landing our beta surfaces this month, and the related issue #2572 took me down a long tangential path for the past week.

@bjwatson

@bjwatson This is something that's my fault, not your fault. There isn't anything you need to fix, we'll always encounter types we don't know about. The proper behavior is just to leave them as Any instances, and I'll make that fix in google-cloud-python

@bjwatson

@bjwatson I can exercise the new logging snippets with GRPC disabled for now.

@dhermes

Ok, thanks @dhermes and @tseaver.

ran into this issue as well. Just to document, you can disable gRPC using

os.environ["GOOGLE_CLOUD_DISABLE_GRPC"] = "true"

I may just end up reverting #2600.

@Fkawala

It's fairly relevant that the only Any in the logging protobufs is LogEntry.proto_payload (which is part of the payload oneof)

Currently digging in to a nice repro provided by @Fkawala, I made it totally distinct of this library here in a gist.

Downgrading logging from 1.0.0 to 0.24.0.

Gracefully continue if LogEntry.proto_payload type URL is not in regi…

Installing google-cloud-logging will work just fine with other packages. For most usage, you can just ignore the google-cloud "uber"-package and focus on the services you actually want to use.

The only place to worry about a version collision is in google-cloud-core, since each package google-cloud-FOO depends on google-cloud-core.

Downgrading logging from 1.0.0 to 0.24.0.

Gracefully continue if LogEntry.proto_payload type URL is not in regi…

When I try to run the code here: https://github.com/proxy/gist.github.com/Fkawala/654093301c47241386b5cd4e02791e56#file-do_log-py

I get an error:

...
...

  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/logger.py", line 302, in log_proto
    client.logging_api.write_entries([entry_resource])
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/_gax.py", line 127, in write_entries
    entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/_gax.py", line 499, in _log_entry_mapping_to_pb
    ParseDict(mapping, entry_pb)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 402, in ParseDict
    parser.ConvertMessage(js_dict, message)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 433, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 539, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse protoPayload field: Can not find message descriptor by type_url: type.googleapis.com/google.appengine.logging.v1.RequestLog..

Does it suppose to work? Or is this feature deprecated?
I couldn't fully understand from the conversation..

Unclear if this should be reopened or is a new issue but I get same error when trying to parse a Protobuf payload written by GCP services: json_format.MessageToJson(payload_pb)

It seems the only way to consume GCP logging outside of the UI is to export to bigquery sink and query from there. Or is logging only meant to work with a logger ?

Btw, disabling gRPC has no effect in this case.

@yiga2

@yiga2 Looks like you opened #5313 to track the parsing issue for protobuf payloads.

Related but #5313 is to request more documentation whereas here is to report an error with MessageToJson.

If the documentation highlights a different method to use, I agree my present comment is not applicable.

@yiga2

@yiga2 json_format.MessageToJson is defined in the protobuf package: please report issues with formatting your particular message type there (I see a couple of open issues related to that function already).

@mikelambert

as an example, here's how i generated my own request_log_pb2.py file based on request_log.prototo parse google.appengine.logging.v1.RequestLog serialized protobufs: https://stackoverflow.com/a/59414384/186123. obviously not ideal, i'd rather use an official version maintained by Google instead, but this is an ok workaround in the meantime.

(this also sounds like @mikelambert's use case in #2674 (comment).

@tseaver

When I try to run the code here: https://github.com/proxy/gist.github.com/Fkawala/654093301c47241386b5cd4e02791e56#file-do_log-py

I get an error:

...
...

  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/logger.py", line 302, in log_proto
    client.logging_api.write_entries([entry_resource])
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/_gax.py", line 127, in write_entries
    entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/_gax.py", line 499, in _log_entry_mapping_to_pb
    ParseDict(mapping, entry_pb)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 402, in ParseDict
    parser.ConvertMessage(js_dict, message)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 433, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 539, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse protoPayload field: Can not find message descriptor by type_url: type.googleapis.com/google.appengine.logging.v1.RequestLog..

Does it suppose to work? Or is this feature deprecated?
I couldn't fully understand from the conversation..

@tseaver Are you able to fix this issue?

Need some help for me, I am facing same issue

Downgrading logging from 1.0.0 to 0.24.0.

Gracefully continue if LogEntry.proto_payload type URL is not in regi…

Navigation Menu

Repository navigation

New 'MessageToDict' implementation breaks with unknown entry types #2674

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

New 'MessageToDict' implementation breaks with unknown entry types #2674

Description

Activity

dhermes commented on Nov 3, 2016

bjwatson commented on Nov 3, 2016

dhermes commented on Nov 3, 2016

tseaver commented on Nov 3, 2016

bjwatson commented on Nov 3, 2016

salrashid123 commented on Feb 6, 2017

dhermes commented on Mar 31, 2017

dhermes commented on Mar 31, 2017

3 remaining items

dhermes commented on Apr 21, 2017

eyalev commented on Aug 29, 2017

yan-hic commented on May 9, 2018

tseaver commented on May 9, 2018

yan-hic commented on May 9, 2018

tseaver commented on May 10, 2018

snarfed commented on Dec 19, 2019

yeruva7 commented on Mar 3, 2021

yeruva7 commented on Mar 3, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions