How do i import data from one collection to another in mongodb?

Lee Theobald

unread,

Feb 20, 2012, 6:16:25 AM2/20/12

to

Hi all,

Hi all.  I was wondering what's the best way to copy items from one collection to another.  As far as I can tell, I have 3 options:

1. Copy via a loop & using db.eval() as suggested via the MongoDB FAQ.
2. Performing a MapReduce and spitting the results out into my new collection (map of "function() { emit(this._id, this ); }", reduce of "function(key, values) { return values[0]; }")
   
3. Performing a MapReduce and saving the results to my collection as part of a finalize function. 
   

What's the best to go for?  #1 (db.eval) is a blocking action so I'd rather not take that option.  #2 does the job but wraps the output in a value object that I don't need/want.  #3 seems like a very roundabout way of doing #1 but at least it doesn't block.

We're using MongoDB to hold all the data we use for analytics.  When a request is made to one of our websites, the data goes into a general "daily_data" collection.  At the end of the day, we copy that days data from this collection into a monthly archive table (e.g. archive_201202).  I'm trying to figure out the best way of performing this copy whilst having the least impact on the system.

Thanks for any input,
Lee

Scott Hernandez

unread,

Feb 20, 2012, 8:07:13 AM2/20/12

to

7 different ways to clone MongoDB connection

• 19th April 2020
• 5 min read
• MongoDB
• 194809 views

There are several ways how to clone a collection within the same or to a different MongoDB database.
We'll measure cloning time, point advantages and disadvantages, compare usability (5-point scale, higher value is better) for each way.
We hope this small research will help to find the fastest way for you.

We'll test performance on replica set cluster in local network. The replica set consists of three MongoDB 4.2 members. Test collection collection1 contains 1 million documents with size about 500Mb.

We'll consider seven different ways:

1. db.collection.copyTo() command
2. db.collection.find().forEach() command
3. db.collecion.aggregate() command
4. mongodump and mongorestore tools
5. mongoexport and mongoimport tools
6. Duplicate collection tool of NoSQL Manager for MongoDB
7. Copy collection to another database tool of NoSQL Manager for MongoDB

The results of our comparison you can find here.

1. db.collection.copyTo() command

Usage - run the command in Shell:

db.collection1.copyTo("collection2")

Execution time: 7m 5s (tested on MongoDB 4.0 cluster)
Pros:
+ Usability: 4.
Cons:
- Can only be run against MongoDB 4.0 or earlier versions.
- Can clone collection to the same database only.
- Very slow.
- Doesn't copy collection properties and indexes.

2. db.collection.find().forEach() command

Usage - run the command in Shell:

db.collection1.find().forEach(
function(docs){
db.collection2.insert(docs);
})

Execution time: 7m 45s
Pros:
+ Usability: 4.
Cons:
- Can clone collection to the same server only.
- Very slow.
- Doesn't copy collection properties and indexes.

3. db.collecion.aggregate() command

Usage - run the command in Shell:

db.collection1.aggregate([{ $match: {} }, { $out: "collection2" }])

Execution time: 15s
Pros:
+ Very fast.
+ Usability: 4.
Cons:
- Can clone collection to the same database only.
- Doesn't copy collection properties and indexes.

db.collection.aggregate() command reference

4. mongodump and mongorestore tools

The mongodump and mongorestore tools are parts of the MongoDB tools package.
You can download the tools package via MongoDB Download Center.

Usage - run the command in command line:

mongodump.exe  --host <host> --port <port> --db test --collection collection1 --out "x:\out"
mongorestore.exe  --host <host> --port <port> --db test --collection collection2 "x:\out\test\collection1.bson"

Execution time: 18s (9s mongodump + 9s mongorestore)
Pros:
+ Very fast.
+ Can clone collection to another database and server.
Cons:
- Usability: 3.

5. mongoexport and mongoimport tools

The mongoexport and mongoimport tools are parts of the MongoDB tools package.
You can download the tools package via MongoDB Download Center.

Usage - run the command in command line:

mongoexport.exe /host:<host> /port:<port> /db:test /collection:collection1 /out:collection1.json
mongoimport.exe /host:<host> /port:<port> /db:test /collection:collection2 /file:collection1.json

Execution time: 44s (27s mongoexport + 17s mongoimport)
Pros:
+ Fast.
+ Can clone collection the a diffretent database and server.
Cons:
- Usability: 3.
- Doesn't copy collection properties and indexes.

6. Duplicate collection tool of NoSQL Manager for MongoDB

Duplicate Collection is a professional feature of NoSQL Manager for MongoDB Pro. It allows to duplicate collection very quickly within the same database.

Right-click on collection1 collection in DB Explorer and select Duplicate 'collection1' Collection... item in the popup menu.

Specify destination collection name, duplication parameters and click Duplicate.

Execution time: 15s
Pros:
+ Very fast.
+ Copies collection properties and indexes.
+ Usability: 5.
Cons:
- Can clone collection to the same database only.
- Available in Pro version only.

7. Copy collection to another database tool of NoSQL Manager for MongoDB

Copy Collection to another Database is a professional feature of NoSQL Manager for MongoDB Pro. It allows to copy one or many collections between databases and servers.

Right-click on collection1 collection in DB Explorer and select Copy 'collection1' Collection to another Database... item in the popup menu.

Specify destination database, additional parameters and click Execute.

Execution time: 35s
Pros:
+ Fast.
+ Copies collection properties and indexes.
+ Can copy collection to another database and server.
+ Can copy several collections at once.
+ Usability: 5.
Cons:
- Can't rename collection.
- Available in Pro version only.

Summary

How do I transfer data from one collection to another in MongoDB?

How do I import collections in MongoDB?

How do I copy an index from one collection to another in MongoDB?