Cara menggunakan mongodb instead of elasticsearch

At an old job, we wanted to migrate our Mongo backend to Elasticsearch. My first task was to find a way to do this elegantly.

Table of Contents

• How do I sync data between MongoDB and Elasticsearch?
• How does Elasticsearch integrate with MongoDB?
• Why use Elasticsearch instead of MongoDB?
• Which is better MongoDB or Elasticsearch?

I decided to use Mongo and Elasticsearch’s Python clients to achieve this.

First, we have to initialize our connection with Mongo and Elasticsearch.

from pymongo import MongoClient
from elasticsearch import Elasticsearch
import os

# Mongo Config
client = MongoClient(os.environ['MONGO_URI'])
db = client[os.environ['MONGO_DB']]
collection = db[os.environ['MONGO_COLLECTION']]

# Elasticsearch Config
es_host = os.environ['ELASTICSEARCH_URI']
es = Elasticsearch([es_host])
es_index = os.environ['ELASTICSEARCH_INDEX']

We can then create this migrate() function that uses Elasticsearch’s helpers API.

We iterate through the collection documents and add them to this actions list.

The key-value pairs in each document will be dumped into Elasticsearch’s _source field.

The _id of each document needs to be removed and moved to Elasticsearch’s _id field to avoid _id conflicts.

from elasticsearch import helpers
import json

def migrate():
  res = collection.find()
  # number of docs to migrate
  num_docs = 2000
  actions = []
  for i in range(num_docs):
      doc = res[i]
      mongo_id = doc['_id']
      doc.pop('_id', None)
      actions.append({
          "_index": es_index,
          "_id": mongo_id,
          "_source": json.dumps(doc)
      })
  helpers.bulk(es, actions)

helpers.bulk() is what will perform the bulk store into Elasticsearch.

If any of the documents in Mongo contain an unserializable object like datetime, we can use a default converter in json.dumps().

import datetime
json.dumps(doc, default = defaultconverter)
def defaultconverter(o):
  if isinstance(o, datetime):
    return o.__str__()

1. Introduction

Elasticsearch is a powerful open source search and analytics engine that makes data easy to explore. It is built on top of the Apache Lucene search library and offers a simple yet powerful interface to perform a wide range of search operations. Elasticsearch is a distributed, scalable, and highly available search and analytics engine. It is used in a wide range of applications including web search, logging, and analytics. Elasticsearch is built on top of the Apache Lucene search library and offers a simple yet powerful interface to perform a wide range of search operations.

2. What is MongoDB?

Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. MongoDB is a cross-platform document-oriented database program. Classified as a NoSQL database program, MongoDB uses JSON-like documents with schemas.

3. What is Elasticsearch?

Elasticsearch is a powerful open source search and analytics engine that makes data easy to explore. It's built on top of the Apache Lucene search library and can be used to power fast and relevant search results for websites and applications. Elasticsearch is also distributed, meaning it can be scaled horizontally to meet the needs of your application.

4. How to import MongoDB data into Elasticsearch

Elasticsearch is a powerful open source search and analytics engine that makes data easy to explore. It can be used to search, analyze, and visualize data from a variety of sources, including MongoDB. To import MongoDB data into Elasticsearch, you need to use the MongoDB River plugin. This plugin allows you to index and search data from MongoDB collections.

5. Conclusion

Elasticsearch is a powerful search and analytics engine that makes it easy to get the data you need. It's also scalable and easy to use, making it a great choice for businesses of all sizes.

by Anurag Srivastava, Mar 9, 2019, 8:20:38 AM | 3 minutes |

In this blog, I am going to explain how you can push your MongoDB data into Elasticsearch. You may be thinking what is the benefit of sending MongoDB data into Elasticsearch so let me explain to you the scenarios where you may want to push MongoDB data into Elasticsearch.

• If you want to apply a robust search for your data.
• You want to collect your data on a central Elasticsearch cluster for analysis and visualization.

Now to push the data into Elasticsearch we need the "logstash-input-mongodb" input plugin for Logstash. So let us see how we can install this input plugin and can configure Logstash for pushing MongoDB data into Elasticsearch.

If you want to know the basics of Logstash then please refer to the "Introduction to Logstash" blog where I have explained the basics of Logstash.

First, we have to log in as root user:

sudo su

Then we can go to the Logstash installation directory (based on your operating system):

cd /usr/share/logstash

Now we can execute the following command:

bin/logstash-plugin install logstash-input-mongodb

We will be getting the following response:

Validating logstash-input-mongodb
Installing logstash-input-mongodb
Installation successful

Now we need to create a configuration file to take MongoDB data as input.

input {
        uri => 'mongodb://usernam::27017/anurag?ssl=true'
        placeholder_db_dir => '/opt/logstash-mongodb/'
        placeholder_db_name => 'logstash_sqlite.db'
        collection => 'users'
        batch_size => 5000
}
filter {
}
output {
        stdout {
                codec => rubydebug
        }
        elasticsearch {
                action => "index"
                index => "mongo_log_data"
                hosts => ["localhost:9200"]
        }
}

After creating the configuration file, we can execute the following command to pull the data.

bin/logstash -f /etc/logstash/conf.d/mongodata.conf

This command will start fetching from the users collection as we have mentioned the name of the collection in the configuration file. This will also create an Elasticsearch index as "mongo_log_data" and will push the MongoDB data.

How do I sync data between MongoDB and Elasticsearch?

To sync the data to ElasticSearch, MongoDB needs to run in replica-set mode. Once the initial sync is completed, it then tails the Mongo oplog(Operation Log) to keep everything in sync in real-time.

How does Elasticsearch integrate with MongoDB?

So here's how you can setup a single node Elasticsearch "cluster" to index MongoDB for use in a NodeJS, Express app on a fresh EC2 Ubuntu 14.04 instance. Make sure everything is up to date. Install NodeJS. Install MongoDB - These steps are straight from MongoDB docs.

Why use Elasticsearch instead of MongoDB?

ElasticSearch is capable to handle queries through REST API and this is its advantage over MongoDB. Flat documents can easily be stored and without degrading the performance of the entire database. In addition to this, ElasticSearch is capable to handle data through filters.

Which is better MongoDB or Elasticsearch?

Elasticsearch and MongoDB are popular document-oriented database. Both are distributed and highly scalable datastores. ... Difference between Elasticsearch and MongoDB..