Using gorm outside grails to import all countries data to mongodb.

 

I am really excited to share this post which makes use of GORM, a very powerfull ORM based on hibernate and groovy as a wrapper over it.

I used gorm for many projects within grails project since grails version 1.x to 3.x. With latest release of grails, gorm can be used as a separate module as a depedency from maven repo.

This blog post is about how we can use gorm within groovy script and import some data to mongodb collection using external web service.

Step 1: Make sure your mongodb instance is up, you can do so by running mongo from your path.

Step 2: Make sure you have jdk and groovy installed on your system.

 

C:\Users\KRISHNA>groovy -v
Groovy Version: 2.4.3 JVM: 1.8.0_31 Vendor: Oracle Corporation OS: Windows 8.1

 

Step 3:  Next run groovyConsole from your cmd which will open a swing based interface where we can write and run groovy script.

Step 4: You can register on www.geonames.org/ to get all the country list. The endpoint to fetch all countries is given below:

 

api.geonames.org/countryInfoJSON?username=your_user_name 

 

Step 5: Final the groovy script which imports country data to mongodb collection named as country.

// https://mvnrepository.com/artifact/org.grails/grails-datastore-gorm-mongodb
@Grapes(
    @Grab(group='org.grails', module='grails-datastore-gorm-mongodb', version='6.1.4.RELEASE')
)


import org.grails.datastore.mapping.mongo.MongoDatastore
import grails.gorm.annotation.Entity
import groovy.json.JsonSlurper

String GEO_URL = 'http://api.geonames.org/countryInfoJSON?username=your_user_id'

String countryJsonString = GEO_URL.toURL().text
def sluper = new JsonSlurper()
def country = sluper.parseText(countryJsonString)
int batchSize = 100
def countryList = []
 country.geonames.each{
     countryList << it
 }
 
int remainingItem = countryList.size() % batchSize 

@Entity
class Country{
String continent 
String capital
String languages
String geonameId
String south
String isoAlpha3
String north
String fipsCode
String population
String east
String isoNumeric
String areaInSqKm
String countryCode
String west
String countryName
String continentName
String currencyCode
}


def initializer = new MongoDatastore(
['grails.mongodb.url':'mongodb://localhost:27017/sampledb'],Country)


def batch = []
countryList.eachWithIndex{cntry,idx->
def c = new Country(cntry)
    batch << c
    if(batch.size() == batchSize || (batch.size() == remainingItem && idx == countryList.size()-1)){
        Country.withTransaction{status ->    
        batch.each{
          it.save()
        }
   }
   batch.clear()
   println "${idx} record processed"
  } 

}
println "record processed"
initializer.close()

 

Lets go through the code line by line so that we can understand how the above script is working. 

From line 2 to 4 we use Grape as a depedency manager which is built into groovy and is based on Ivy and uses maven for depedency resolution. Here we add Gorm 6.1.4 version to use it in the script.

Line 13 is a groovy way of making http request which holds the json string as a response.

Line 14 uses groovy in built JsonSluper which can parse json string, def is a dynamic way of declaring variables in groovy.

Line 24 @Entity is a gorm way of marking a class as an Mongo Entity.

Line 45 is a connection to mongodb along with collection binding.

Line 52 is creating country instance by passing cntry as a map returned from jsonParser.

Here the data is saved to collection in a batch size of 100 for better performance.

Groovy is a very  powerful language that can be used along with java for making code more simpler.