Migrating Data from Firestore to MongoDB¶

In this article, we'll go through the steps on how to:

Export Firestore Data
Download the Exported data from the Bucket
Convert the Firestore export output to JSON
Import the JSON data into MongoDB

Prerequisite 1: Setup a GCloud container on MedStack Control¶

In order to interact with Google Cloud, you will need to run gcloud-sdk container on MedStack.

Create a new Service in MedStack Control and use google/cloud-sdk as the image for the container and mount the mongodb data volume to this container

Prerequisite 2: Install Firestore export to JSON converter inside GCloud container¶

Once the container is running, shell into the container and follow the steps below to install the script that will be used to convert Firestore's export to JSON.

git clone https://github.com/labbots/firestore-export-json.git
apt-get install -y libffi-dev python3-venv
cd firestore-export-json
python3 -m venv venv
source ./venv/bin/activate
make install

Prerequisite 3: Create a MongoDB service on MedStack Control¶

Create a MongoDB service on MedStack Control.

Exporting Firestore Data¶

There are 2 ways to export data from Firestore database: 1) Google Cloud Dashboard and 2) GCloud CLI.

Google Cloud Dashboard¶

Follow the steps below to export Firestore data using the Google Cloud Dashboard.

Go to the Import/Export page on GC Dashboard.
Make sure the correct project is selected.
Click on the Export button
Either Select an Individual Collection or the Entire Database options to Export
Select a Google Cloud Bucket as a destination for the exported content
Click Export to initiate the data export.
Once the export is complete, you should see an entry in the Recent import and exports section.

GCloud CLI¶

Another method for creating an Export of the Firestore data is via the usage of Google Cloud CLI (GCloud CLI).

Note: You will need to have the GCloud SDK installed on your workstation or you can use the google/cloud-sdk container from Prerequisite 1.

Google Cloud Authentication¶

Login to Google Cloud Project using the command below.
```
gcloud auth login
```
Copy the URL from the Terminal and open it in your browser.
Select the account that is associated to the project
Click on the Allow Button to give Google Cloud SDK access to you account
Click on the Copy button to copy the authorization code and Paste the Authorization code in the Terminal to complete the authentication.
You should now be successfully login

Export Firestore Data¶

Once you have logged In, Run the command below to list all the project and copy the project ID of the project you wish to export:
```
gcloud projects list
```
Once you have the project ID, run the command below to set the default project
```
gcloud config set project {project_id}
```
Finally you can run the export command and substitute the project ID in the URL:
```
gcloud firestore export gs://{project_id}.appspot.com
```
Once the export is complete you should get the metadata of the export. Copy the value of outputUriPrefix. This URI represents the folder location in the Google's Cloud Storage Buckets where the export is stored.

Downloading Data from Cloud Storage Buckets¶

Once the export is complete, the exported data is stored inside the Google's Cloud Storage Buckets.

Method 1¶

You can either download the exported content by following the steps below:

Select the export directory,
Click on the Download button,
Copy the gsutil command and
Run the command in the gcloud container from Prerequisite 1.

Method 2¶

You can also download the exported directory by running the command below directly.

Note: You will need 2 pieces of information prior to running the command below: 1) outputUriPrefix from the metadata output and 2) Directory Path where the Volume is mounted inside the container.

gsutil -m cp -r {outputUriPrefix_Value} {Volume_Mount_location}

You can also get a list of all the exports by running the following command:

gcloud firestore operations list

From output you can locate the desired export and grab the value of the outputUriPrefix field and use the above gsutil command to download the exported files to the specified location.

Converting exported output file to JSON¶

Once the exported data is downloaded from Google Cloud Storage, the exported data will need to be converted to JSON format in order for it to be imported into MongoDB.

Use the commands below to convert the exported data to JSON.

Note: In order for the command to work, you will need to install the Firestore export to JSON converter. See Prerequisite 2 for more information.

Go to the directory where the git project was downloaded and go into the project directory
```
cd firestore-export-json
```

Activate the python virtual environment

source ./venv/bin/activate

3) Execute the command with absolute path to downloaded firestore export and absolute path for the output JSON file.

fs_to_json path_to_source_dir path_to_destination

Example:

fs_to_json /home/2022-09-01T20\:27\:48_62024/all_namespaces/all_kinds/ /home/.

Importing JSON Data into MongoDB¶

Once you've exported and converted your Firestore data into JSON format, the next step is to import it into MongoDB using the mongoimport tool.

Installing Mongo Tools¶

Before you can proceed with the JSON import, you need to install the MongoDB tools. Use the following command to install them:

sudo apt install mongo-tools

Importing JSON Data with mongoimport¶

To import your JSON data into MongoDB, use the following command:

mongoimport --uri <MONGODB_CONNECTION_URI> \
  --collection <COLLECTION_NAME> --type json --file <FILENAME.json>

Replace the placeholders with your specific details:

<MONGODB_CONNECTION_URI>: The URI to your MongoDB instance. It includes your username, password, host, port, database name, and authentication source (admin).
<COLLECTION_NAME>: The name of the MongoDB collection where you want to import the data.
<FILENAME.json>: The name of your JSON file containing the data.

Here's an example command:

mongoimport --uri mongodb://root:pass12345@localhost:27017/firestore?authSource=admin \
--type json --collection userProfile --file firestoreData.json

After running this command, you should see output indicating that the import was successful, along with the number of documents imported.

2022-09-22T23:24:18.800+0000 connected to: mongodb://[**REDACTED**]@localhost:27017/firestore?authSource=admin
2022-09-22T23:24:18.806+0000 6 document(s) imported successfully. 0 document(s) failed to import.

By following these steps, you can efficiently import your JSON data into MongoDB, making it available for your applications and further analysis.