A Cursor of What?

In a typical MongoDB driven .NET application, we are not going to be using a cursor like we would if this were a sql database. We don’t want rows and we don’t want to read lines. We want objects. We’re greedy like that… and it will save us a lot of time.

First a reference to the documentation. If you want more detail, this is where you need to go.

To get a cursor of objects, you need to have saved (serialized) some objects previously. See our first MongoDB .NET tutorial for an example of how to do this. Picking up where that left off, let’s say we have some instances of our Car class saved in MongoDB.

MongoDB Cursor of Cars

Of course we need a connection to the database and we need a reference to the CarCollection, something like this:

server = MongoServer.Create("mongodb://localhost")
db = server("dbMyDatabaseName")
cars = db("CarCollection")

Now we can build a query and get a cursor. First let’s do it in two steps for clarity.

dim qry = Query.EQ("fast", True)
dim mcCars = cars.FindAs(Of Car)(qry)


At this point the query has not executed. It will lie in wait until we actually enumerate the cursor (mcCars).

For Each c as Car in mcCars
   Console.WriteLine(“Found fast car named: ” & c.name)
Next

The object conversion was implicit in the drivers behavior. We asked for cars where the “fast” property in our class definition was set to True, and we got a strongly typed collection of Cars in our cursor.

For convenience, a shorter way to ask is:
dim mcCars = cars.FindAs(Of Car)(Query.EQ("fast", True))

Ordering

Now that we know how to obtain a cursor, what are some options for tweaking it? We can do pretty much anything that SQL can do.

Bear in mind we are working with objects, not numbers and strings. So when you query this way there is no exact parallel for SUM, GROUP BY, etc. That’s a topic for another day.

Instead of ordering results in the query, we can do it by setting properties on the cursor. For example, if we want all Cars in the collection, ordered alphabetically by name:

dim mcCars = cars.FindAllAs(Of Car)()
mcCars.SetSortOrder(MongoDB.SortBy("name"))


We can add other options too:

mcCars.Limit(100) ' get only the first 100 results
mcCars.SetFlags(NoTimeout) ' never timeout
mcCars.Skip(10) ' skip the first 10 results


TIP: Be careful with the NoTimeout option. I needed it for a long running operation that takes many hours to iterate over a cursor, so I had no choice. The thing to know is that your cursor will timeout by within 15 minutes (the default in version 1.3 of the official MongoDB .NET driver) from the time you being iterating. If you set the NoTimeout flag, it will never timeout and if your application dies the server will not release the cursor until the database restarts. You can easily build up a large pile of wasted resources this way if you’re not careful.

Or get the result count:
Console.WriteLine("Found " & mcCars.Count & " cars.")


That’s basically it. Lot’s of advanced options can be found in the driver documentation.
Next up: More Queries.

Get the MongoDB .Net Driver

http://github.com/mongodb/mongo-csharp-driver/downloads
First install the latest driver for your platform.

Create a .Net Project and Add the Reference to Visual Studio

In Visual Studio, click Project, Add Reference, and then select the .Net tab. You will probably find MongoDB.Driver and MongoDB.Bson near the bottom.  Select them both. If you don’t see them here you can also browse directly to the dll files installed by the driver installation routine, probably in \Program Files\CSharpDriver 1.3\.

Connect

In your vb code file, add the imports (‘Builders’ is needed for queries):
Imports MongoDB.Bson
Imports MongoDB.Driver
Imports MongoDB.Driver.Builders

Define some variables that can be used by all of our methods:
Dim server As MongoServer
Dim db As MongoDatabase
Dim cars As MongoCollection

Create an actual server connection to localhost when we initialize the program in form.Load or wherever (of course, you need to be running a MongoDB server on localhost. If not, set one up first or substitute another DNS name or IP address for ‘localhost’):

server = MongoServer.Create("mongodb://localhost")
db = server("dbMyDatabaseName")
cars = db("CarCollection")

Here we are connecting to the server, then a database named dbMyDatabaseName, then inside that to a collection named CarCollection. The act of connecting to a database or collection will cause it to be created if it doesn’t exist.

// Begin Quick Explanation

Skip this if you already know how MongoDB works.
It isn’t like a relational database where rows of information are stored in tables, which in turn are stored in the database. It works like this:

• The basic unit of data is the name / value pair.
• The name is a string and the value can be anything: a string, integer, date, array, etc.
• One or more name / value pairs are stored together in a document.
• Related documents are stored in a collection.
• A database is a group of collections.

Most often, you will probably be storing instances of your classes as documents, which is exceptionally easy to do with MongoDB and the .Net driver. This is great for developers because you can code a lot faster with no need for a3rd party object relational mapper (ORM) or a lot of functions just for translating objects to SQL statements. For example, if I need a database to store information about cars, I would do it like so (exact syntax coming up):

• Create an instance of car
• Set some attributes
• Save it to MongoDB

Now I will have a “document” that contains the attributes of my car object, which I can easily retrieve and change.

// End Quick Explanation

Now that we have a database connection and a collection to put things in, we can write some data. Assume we have a simple car class that creates a BSON id whenever a new Car is created.

Class Car
Imports MongoDB.Bson
Public Property _id as ObjectID
Public Property name As String = ""
Public Property topSpeed As Integer = 0
Public Property fast As Boolean = False

Public Sub New()
   _id = new ObjectID()
End Sub

End Class

We can now instantiate a car and save it very easily because MongoDB and the VB.NET driver will automatically serialize your object:
dim car1 As New Car
car1.name = "Bob"
car1.topSpeed = 100
car1.fast = True

cars.Save(car1)

TIP: Normally we don’t really want to work with “documents”, we want to work with strongly typed objects (instances of our classes that have been saved to the database). Right in the query syntax we specify that we want an object returned by the query, not a document that we then have to “translate” into one of our objects. See the following query!

To search for the car by name:
dim somecar = cars.FindOneAs(Of Car)(Query.EQ("name", "Bob"))

This query will retrieve the first car named Bob. If there are no cars name Bob, then somecar will be null (Nothing). Remember to check:

If not somecar Is Nothing Then
   console.Writeline("We found a car")
Else
   console.Writeline("Query matched nothing...")
End If


Update and Save

Updating is as simple as inserting. Because we retrieved “Bob” as an instance of Car, we just make our changes and save. After we retrieve somecar:

somecar.topSpeed = 110
cars.Save(somecar)

The magic is in the _id property. If the MongoDB driver sees that a field named _id exists and is set, it will automatically update an object with a matching _id (in this case, Bob) instead of creating a new one. Otherwise, it would have no way of knowing if this was a new Car, also named Bob, or an update to the existing Car named Bob.

TIP: If the _id’s match it is an update. This is sometimes called an “Upsert”, meaning: Update if it exists, Insert if it doesn’t. This is the default behavior of the “CollectionName”.Save function in the official MongoDB .NET driver. It saves you a lot of time.

An easy way to explore the database and see your changes is to use the excellent tool MongoVue. For those who prefer a GUI or haven’t yet spent any time learning the MongoDB command syntax, it’s a huge timesaver. We use it in our office every single day.

MongoDB enables rapid data-driven application development. Since we had four days to get it done we went MongoDB and finished the app in record time.

Our partners came to us with a problem. They had a customer with a pile of paper medical records and medical staff in multiple locations who needed real time access. The records had to be digitized and captured in a database along with basic patient information, and we needed a delivery framework to allow doctors with iPads to securely access the data from anywhere.

And only one last requirement… They needed to go live with this system in four days! Had we been stuck with a SQL back end, this would have been impossible. But with MongoDB and the automatic object serialization built in to the official MongoDB .Net driver we can pull this off on time and on budget.

First we modeled the data in our .Net classes. This was pretty simple as we only needed to represent a person with some basic properties, plus any files associated with them. We built a GUI that would allow us to create instances of our people and set their properties. And then we built the data access / persistence layer by connecting to a MongoDB instance and persisting our instance to the database by calling collectionName.Save(myPerson).

To get that person back, we simply ask:
dim person1 = collectionName.FindOneAs(Of Person)(Query.EQ(“_id”, “SomeID”))

Or by Social Security number:
dim person1 = collectionName.FindOneAs(Of Person)(Query.EQ(“ssn”, “555-55-5555″))

That’s It

Sometimes building a data access layer can be time consuming, or hard, or full of those little details that make you crazy and consume vast quantities of development time. But not this time. MongoDB has a .Net driver that automatically serializes an object into a BSON Document. For us that means we can code away and when the time comes to CRUD (Create, Read, Update, Delete), it pretty much just happens without any extra effort on our part.

Is that a big deal?

We built this entire project in about the same amount of time we spent on the data access layer of our last project. It was slightly larger, but the huge time savings captured by not having to break your objects into tiny little SQL pieces (or configure a third party ORM to do it) is frankly awesome. It cannot be overstated. An object maps pretty naturally to a document, and all the basic types are supported (strings, integers, lists / arrays, dates, binary, etc).

In our next series of posts we’ll break down how we built it. Hopefully this will assist any developers out there who are interested in MongoDB but want to see a real life example before taking the plunge and learning a new technology.

Many data-driven applications and websites could dramatically improve performance by optimizing MySQL.

The developers who built ROS.com did a nice job on the php, but the performance of the database and underlying server was never addressed. A MySQL and PHP upgrade brought the problems into sharp focus and Hawkins IT Group was selected to help. In a few hours we analyzed the performance of the site and the underlying MySQL database and had a list of changes ready.

• Query cache needed to be turned on
• Key buffer needed to be large enough to hold the MyISAM indexes in memory
• Innodb buffer needed to be larger to hold the InnoDB clustered indexes in memory
• Table cache needed to be much larger
• Query Analysis: Indexes added to eliminate joins without indexes

And the result was great! Some pages that required multiple queries to load were previously taking over 30 seconds to complete. This is now down to about 2 seconds. Our MySQL database performance is vastly improved and the slow query log–set to record any queries over 1 second in duration–is still empty. All queries complete in less than one second.

Optimization HowTo

If you need a performance boost out of your MySQL database, start by following these steps:

1. Enable the slow query log by adding these lines to your my.cnf file:
log-slow-queries = /var/log/slow-queries.log
long_query_time = 1

2. Acquire and run some basic analysis tools:
My personal favorite is mysqltuner.pl. Google for it, run it, and get some numbers. The key statistics are the ones mentioned above. Research them and make them right for your database by setting the needed variables in my.cnf.

3. Check the slow query log. Pull out some slow queries that seem to recur and run them manually. See if you need to add any indexes. Run the queries again to see if it helped (but disable query caching temporarily so you don’t get a false result).

4. Try a round of optimizations and restart MySQL. After letting the caches warm up, run the analysis tool again and see if you’ve made any improvements.

5. Repeat!

This will get you started in the fundamentals of MySQL optimization. If you want professional help, please contact us and we’ll work with you on the details.

CLRC needed an application to track all the customized information they need to store for their member libraries. After a bit of development, we provided our first draft of the CLRC Database for testing. Now in round 3 of customizations, we’ve been upgraded to live usage at the customer site. Hopefully this is our last round of changes and CLRC can begin enjoying the fruits of their vision for better library council contact management.