Elasticsearch Masterclass: ES|QL, Vector, and More

Elasticsearch 7 : This section of the course will give you the clear idea about the elasticsearch and how elasticsearch is processing the data. and you will get clear idea about how elasticsearch is differ from database system.

Elasticsearch is a distributed document store. Means, once the document has stored in Elasticsearch, it can be retrieved from any node in the cluster. All other languages can communicate with Elasticsearch over port 9200 using a RESTful API, accessible with your favorite web client. You can even talk to Elasticsearch from the command line by using the curl command. The hostname of any node in your Elasticsearch cluster, or localhost for a node on your local machine. And then port, The port running the Elasticsearch HTTP service, which defaults to 9200.

Elasticsearch 7 : This section of the course will provide you the idea of CRUD operation in elasticsearch.  Its basically how elasticsearch indexing the data, how elasticsearch updating existing data in elasticsearch cluster and how elasticsearch delete the data from elasticsearch cluster and Obviously searaching the data in elasticsearch through the elasticsearch cluster.

In Elasticsearch, a document belongs to a type, and those types live inside an index. An elasticsearch cluster may contain multiple indexes. Its Overview of elasticsearch. Whenever we index this document in the first request, the title was Introduction to elasticsearch. And In the 3rd request, we have updated the title to this document to Overview of elasticsearch.

Elasticsearch 7 :  This section of the course, provide you the search operation in elasticsearch. The reason to store the data in elasticsearch cluster is for searching it back. And elasticsearch is created for that only.

Just imagine, you went to one website which contains blogposts across almost all the technologies. If you are interested only in the blog post related to elasticsearch, then you will probably type the word elasticsearch in the search box on the website. You expect the website should display only the blog posts related to elasticsearch right. Along with that, you expect, the posts which are more relevant for the word "elasticsearch" should be displayed at the top of the page instead of random order. If you search for text "Indexes in elasticsearch," You may end up with lots of posts related to elasticsearch. But you will expect the blog post that is talking about the concept of indexing in elasticsearch should display at the top of the result. Am I correct?It sounds too complicated to implements this right; don't worry about it. Elasticsearch made up for this purpose. It provides a rich API for searching through data. By this way, elasticsearch will take care of everything under the hood. You just to focus only on what you need. Let's get started with searching.

Elasticsearch 7 :  This section of the course, provide you the clear idea about the elasticsearch cluster. What what is happening inside the elasticsearch  cluster and elasticsearch  node.

In elasticsearch, All the data lives inside a cluster must belong to any index. An index is like a database in a relational database; Its the place we store related data. So whenever you index the document into the elasticsearch, you have to specify where your document should be stored. Means you have to mention the index during insert the document into the elasticsearch. This index is the logic representation in the Elasticsearch environment. But it's completely different from how elasticsearch sees these indexes in background. Elasticsearch divides index into shards that holds just a slice of all the data in the index. In reality, an index points to one or more physical shards. Don't get confused with index and shards. If you are index a document into an elasticsearch index, it means you are indexing documents into shards. Shards are nothing, but it's a Lucene index. So an Elasticsearch index is made up of multiple Lucene indexes. Logically it's okay right because Elasticsearch uses Apache Lucene as its core library to index your data. So when you index the documents into the elasticsearch index, these documents are indexed in multiple shards that are Lucene indexes.

Elasticsearch 7 :  This section of the course, provide you the clear idea about the components of an elasticsearch index. Every elasticsearch index has settings and mappings to manage the data inside the elasticsearch  cluster.

An index that we have created in previous sections. Whenever you index the documents into elasticsearch, And the specified index already available in elasticsearch, then the documents are indexed inside that. Suppose, the index, it's not available already, Elasticsearch itself will create an index for us. You might remember that, When we index our first document into the blogposts index, that index was not available already. So elasticsearch has created it for us. Let's have a look at the index structure of the blogposts index, which is created by elasticsearch. 

Elasticsearch 7 : This section of the course, provide you the clear idea about the analysis process while we index the data in to the elasticsearch cluster. Every data or documents we sent to elasticsearch will be send to analysis process and finally stored in elasticsearch  cluster.

We have seen, how to index the data into elasticsearch, and also we have seen, how to search for data within it. In this video, we are going to see, How elasticsearch process the data when you push the data into elasticsearch. If you index any document into an elasticsearch, the documents are sent to the analysis process. An analysis is a process of converting text, like the body of a document into tokens or terms. Later, these terms are added to the inverted index for searching. With the help of these inverted indexes, Elasticsearch archives very fast full-text searches. So whenever you index a document into elasticsearch, It goes through a number of steps for every analyzed field before the document is added to the index.

Elasticsearch 7 :  (Part 1) This section of the course, provide you the search operation in elasticsearch. The reason to store the data in elasticsearch cluster is for searching it back. And elasticsearch is created for that only.

Just imagine, you went to one website which contains blogposts across almost all the technologies. If you are interested only in the blog post related to elasticsearch, then you will probably type the word elasticsearch in the search box on the website. You expect the website should display only the blog posts related to elasticsearch right. Along with that, you expect, the posts which are more relevant for the word "elasticsearch" should be displayed at the top of the page instead of random order. If you search for text "Indexes in elasticsearch," You may end up with lots of posts related to elasticsearch. But you will expect the blog post that is talking about the concept of indexing in elasticsearch should display at the top of the result. Am I correct?It sounds too complicated to implements this right; don't worry about it. Elasticsearch made up for this purpose. It provides a rich API for searching through data. By this way, elasticsearch will take care of everything under the hood. You just to focus only on what you need. Let's get started with searching.

Elasticsearch 7 :  (Part 2) This section of the course, provide you the search operation in elasticsearch. The reason to store the data in elasticsearch cluster is for searching it back. And elasticsearch is created for that only.

Just imagine, you went to one website which contains blogposts across almost all the technologies. If you are interested only in the blog post related to elasticsearch, then you will probably type the word elasticsearch in the search box on the website. You expect the website should display only the blog posts related to elasticsearch right. Along with that, you expect, the posts which are more relevant for the word "elasticsearch" should be displayed at the top of the page instead of random order. If you search for text "Indexes in elasticsearch," You may end up with lots of posts related to elasticsearch. But you will expect the blog post that is talking about the concept of indexing in elasticsearch should display at the top of the result. Am I correct?It sounds too complicated to implements this right; don't worry about it. Elasticsearch made up for this purpose. It provides a rich API for searching through data. By this way, elasticsearch will take care of everything under the hood. You just to focus only on what you need. Let's get started with searching.

Elasticsearch 7 :  This section of the course, provide you the idea about the score and relevancy of the elasticsearch document

An elasticsearch query is different from regular SQL queries. Obviously, elasticsearch ability to handle full-text searches and other than that, An elasticsearch has the ability to assign relevancy, or we can call it as the score to a document. With the help of this score, you know how relevant the document is to the original query. When the users type a query into a search box on a website, they expect to find not only results matching their query along with that; they will also expect those results ranked based on how closely they match the query's criteria. By this way, Elasticsearch is quite flexible when it comes to determining the relevancy of a document. But it may make sense too. First, we think about documents matching queries in a binary sense, I mean either "Yes, it matches" or "No, it doesn't match,". So elasticsearch, reduce the documents by applying the boolean test. And then, it makes much more sense to think about documents matching in a relevancy sense. Now it's more logical to say that document 1 is a better match for a query than document 2. For example, when you use your search engine to search for "elasticsearch," it's not enough to say that a particular page contains the term and that's why it matches; instead of that, you want the results to be ranked according to the best and most relevant results. The scoring of a document is determined based on the field matches from the query specified and any additional configurations you apply to the search.

Starting with Elasticsearch 8.0, native support for vector search enables powerful similarity-based querying using high-dimensional dense vectors. This section explains how vector search works, what Approximate Nearest Neighbor (ANN) is, and how to use dense_vector fields and knn queries to implement use cases like semantic search and recommendations. You'll also understand how Elasticsearch’s ANN engine compares with traditional search and what performance benefits it brings through HNSW-based indexing.

Elasticsearch 8+ supports semantic search using NLP models and vector embeddings. This section explains how to index and query using dense vectors, run native inference with models like BERT, and build search systems that understand meaning rather than just keywords.

ES|QL (Elasticsearch Query Language), introduced in version 8.15 and matured in 9.0, brings a powerful pipelined syntax for querying Elasticsearch. It enables SQL-like, readable commands with support for filtering, transformation, aggregation, and even lookup joins across indices. This section covers the syntax, capabilities, and practical use cases of ES|QL and shows how it differs from traditional Query DSL, making data exploration and analysis faster and more intuitive.

Elasticsearch 7 :  This section of the course, provide you the idea about statistics and aggregation of the data in elasticsearch.

So far, we have seen how to index the documents into elasticsearch and how to search the documents based on the keyword we are looking for. In the elasticsearch cluster, we may have millions or trillions of documents, and the user wants to find the most relevant matches for specific keywords. Users may not always be looking for a specific result. I mean, they are not always interested in a particular document. Instead, they want to get statistics from a set of documents. These statistics may be helpful to showcase the revenue report, trends for the different products in the market, or the number of unique visitors in the blog, something like that. Aggregation in elasticsearch helps us to implements the statistic over the data that we have in the elasticsearch.

Elasticsearch 7 :  This will provide you an idea about processing the data in elasticsearch. It means push the data to the elasticsearch from various source system through logstash. 

how we feed the data in to elasticsearch? By directly sending a request with a JSON object to the elasticsearch cluster. This JSON object indexed as a document into elasticsearch. Am I correct? We knew that, elasticsearch has the ability to handle a huge amount of data. It's not designed to work with 100 or 1000 JSON objects. So in real-time, we may get data from different sources like database, application log, or from some other systems.  Now we have to feed these data to the elasticsearch cluster. But how to do that? The elastic stack provides the tool called logstash to pull the data from the source system and push it into the elasticsearch.