Feature Request: Support other database such as clickhouse #2158

Activity

1. skydoorkai commented on Jul 15, 2020

   

   skydoorkai

   on Jul 15, 2020

   Last edited by skydoorkai

   Collaborator

   More actions

   In the current ElasticDL version, one way to support clickhouse or MySQL is using custom_data_reader_params.
   Shard is defined as (shard_name, start_index, num_record).
   Task is defined as (shard_name, start_index, end_index)
   Thus, need to put all required info to access clickhouse or MySQL in shard_name, which is a string.
   Also need to implement data reading in read_records using (shard_name, start_index, end_index).

   https://github.com/sql-machine-learning/elasticdl/blob/develop/docs/designs/custom_data_source_support.md

   We are working on supporting more general data sources.