Add Elasticsearch Capabilities to Application #60

Activity

1. npendery

   changed the title [-]Add Elasticsearch to Application[/-] [+]Add Elasticsearch Capabilities to Application[/+] on Aug 10, 2016
2. joshsmith commented on Aug 10, 2016

   

   joshsmith

   on Aug 10, 2016

   Contributor

   More actions

   To clarify a little here, the Rails implementation using searchkick looks something like this:

   class Skill < ApplicationRecord
     searchkick match: :word_start, searchable: [:title]
   
     def self.autocomplete(query)
       results = search query
       results.take(5)
     end
   end

   class SkillsController < ApplicationController
     before_action :doorkeeper_authorize!, only: [:create, :update]
   
     def index
       if search_params.present?
         skills = Skill.autocomplete(search_params)
       end
     end
   end

   Thus a query like rub should provide results like Ruby and Ruby on Rails. Similarly, a query like rab would also provide Ruby and Ruby on Rails (assuming no other relevant results), because the fuzziness of the query allows for some fat-fingering on the user side.

   I had done some work awhile back to do this manually, and had come up with this:

     def self.query(query)
       terms = query.split(" ")
       terms = terms.map do |term|
         { term: { title: term } }
       end
       results = search query: {
         bool: {
           should: terms
         }
       }
       results.records.take(5)
     end
   
     settings index: {
       number_of_shards: 1,
       number_of_replicas: 0,
       analysis: {
         filter: {
           autocomplete_filter: {
             type: "nGram",
             min_gram: 1,
             max_gram: 3
           }
         },
         analyzer: {
           autocomplete: {
             type: "custom",
             tokenizer: "standard",
             filter: [ "lowercase", "autocomplete_filter" ]
           }
         }
       }
     } do
       mappings do
         indexes :title, analyzer: "autocomplete", search_analyzer: "standard"
       end
     end

   Was not able to get much of it to work, though.

   We should look more deeply into the code behind searchkick and figure out precisely what our working code in searchkick is doing behind-the-scenes in ElasticSearch itself.
3. joshsmith

   added

   Priority: High

   on Aug 12, 2016
4. joshsmith

   added

   L

   hacktoberfest

   ask hn

   on Sep 29, 2016
5. joshsmith

   removed

   hacktoberfest

   ask hn

   on Nov 4, 2016
6. paulsullivanjr commented on Nov 16, 2016

   

   paulsullivanjr

   on Nov 16, 2016

   Contributor

   More actions

   I wanted to have a PR ready by Monday night but didn't have enough time. I've been using the tiresx library. It wasn't to hard to get a simple example working but as I tried to add analyzers I ran into issues. Not huge issues but it did slow me down and there almost no documentation on the library. If time is allowed I almost think it's worth writing a module to handle instead of using a library. The API is well documented and since it's being used in one controller it should be pretty easy to do. Let me know your thoughts.

   Also I did my development outside of the API code. I need a better understanding of how elastic search works within docker. I noticed in the documentation that the default port used by elasticsearch is blocked.

   Thanks
   Paul
7. joshsmith commented on Nov 18, 2016

   

   joshsmith

   on Nov 18, 2016

   Contributor

   More actions

   @paulsullivanjr sounds good! Let me know if you want any help on getting the Docker side up and running.

   If you think calling out to Elasticsearch is easier via a module, then I'm happy to take that approach. It seems reasonable to me, honestly.
8. paulsullivanjr commented on Nov 19, 2016

   

   paulsullivanjr

   on Nov 19, 2016

   Contributor

   More actions

   @joshsmith Sounds good. I'll get started on the development. I'll touch base with you about the docker stuff when I'm ready for it. Thanks.
9. joshsmith commented on Nov 19, 2016

   

   joshsmith

   on Nov 19, 2016

   Contributor

   More actions

   Okay thanks. I'll be around if you need me.
10. paulsullivanjr commented on Mar 11, 2017

    

    paulsullivanjr

    on Mar 11, 2017

    Contributor

    More actions

    After talking to JoshSmith following suggestions were made for the next set of changes to this feature:

    1. Adding a skills/search endpoint for searching, handled directly in SkillController
    2. Endpoint point needs to return a collection of Skill structures, need to research how to create from elasticsearch results. The Elasticsearch indexes will need to contain the Skill records from the database including id's.
    3. Keep an ELASTICSEARCH_URL in .env
11. joshsmith commented on Mar 14, 2017

    

    joshsmith

    on Mar 14, 2017

    Contributor

    More actions

    @npendery out of curiosity do you know how to turn Elasticsearch results into a SQL query that fetches those records? I'm wondering if we want to do some sort of IN query or something. Idk the common happy path.
12. joshsmith

    removed

    Size: L

    on Sep 28, 2017