Repository navigation
Issue uploading from file #2098
Description
There seems to be a fairly major issue with the upload_from_file method.
from gcloud import bigquery
from oauth2client.service_account import ServiceAccountCredentials
from gcloud.bigquery import SchemaField
from gcloud.bigquery.job import CreateDisposition, WriteDisposition
import pprint
credentials = ServiceAccountCredentials.from_json_keyfile_name('keys/key.json')
client = bigquery.Client(project='{{ project_name }}', credentials=credentials)
# datasets, next_page_token = client.list_datasets()
#
# for dataset in datasets:
# pprint.pprint(dataset.name)
# schema = [
# SchemaField('full_name', 'STRING', mode='required'),
# SchemaField('age', 'INTEGER', mode='required')
# ]
#
# dataset = client.dataset('test')
# table = dataset.table(name='test_table', schema=schema)
# table.create()
dataset = client.dataset('test')
table = dataset.table(name='test_table')
with open('data/data.json', 'rb') as json_data:
# pprint.pprint(json_data.read(1))
job = table.upload_from_file(json_data, source_format='NEWLINE_DELIMITED_JSON', encoding='UTF-8', create_disposition=CreateDisposition.CREATE_NEVER, write_disposition=WriteDisposition.WRITE_APPEND)
pprint.pprint(job)We get the following output.
$:/mnt/hgfs/user/Desktop/bqtest# python insert.py
Traceback (most recent call last):
File "insert.py", line 32, in <module>
job = table.upload_from_file(json_data, source_format='NEWLINE_DELIMITED_JSON', encoding='UTF-8', create_disposition=CreateDisposition.CREATE_NEVER, write_disposition=WriteDisposition.WRITE_APPEND)
File "/usr/local/lib/python2.7/dist-packages/gcloud/bigquery/table.py", line 913, in upload_from_file
return client.job_from_resource(json.loads(response_content))
File "/usr/local/lib/python2.7/dist-packages/gcloud/bigquery/client.py", line 119, in job_from_resource
config = resource['configuration']
KeyError: 'configuration'
Activity
dhermes commentedon Aug 12, 2016
Whoa! Is that consistent? The response not containing
configurationseems like a deal-breaker.Could you run
and actually inspect the contents of
resourceon a failure?cwbeck commentedon Aug 15, 2016
@dhermes the reason seems to be due to lack of schema. I made the assumption schema would be fetched automatically into the model, but it appears not to be the case.
Passing in schema of the table fixes the issue. Example given in https://googlecloudplatform.github.io/gcloud-python/stable/bigquery-usage.html shows the schema being added, but I assumed here this was only needed for
create_disposition='CREATE_IF_NEEDED'dhermes commentedon Aug 18, 2016
@jonparrott Wasn't there a related issue to this?
theacodes commentedon Aug 18, 2016
I filed #2132 which has a bit more information on how to correctly solve this - we should check the response of the initial upload request and raise an exception if needed.
tseaver commentedon Sep 6, 2016
@cwbeck Let's track this on #2132.
feat: Add support for reading ClientLibrarySettings from service conf…
fix: Throw type error for incomparable join keys (#2098)