Geo-tagging means associating something - in this case, photos - with its relevant geography - in this case, where the photos were created. This is done by matching the date and time an exposure was made against GPS data. Of course there are many ways to skin a cat, but Exif Harvester does some of its work in downright unique ways.

Better Geo Data

Some of what Exif Harvester does to improve the quality of its data are:

Camera offset - GPS units always have exactly the correct time;  this is how GPS works.  Your camera may not have the right time, however.  Mine is certainly off, and after I set it, it eventually drifts a bit.  A large time difference would yield a large geographic error, if time stamps were simply matched up.  Instead, when you download and tag your photos, you can tell Exif Harvest how far your camera’s clock is from the GPS clock, and this offset will be honored when your photos are geo-tagged.

Point averaging - A common pitfall to geo-tagging software is that most programs simply find the nearest point, and assign it to your photos.  Exif Harvester is a bit more subtle:  if no point in your GPS data exactly matches the time stamp from an image ( adjusted for the camera offset ), the closest points before and after the photo was created, and calculate a weighted average from them.  For example, it might take half an hour to fly from Seattle to Vancouver;  if you shoot a photo ten minutes into the trip, it will be geo-coded to appear 1/3 of the way between these cities, rather than in either of them or halfway between.

Some Guesswork, Too

While the latitude, longitude, and elevation values are discovered in a very robust way, Exif Harvester will attempt to guess which direction the camera may have been facing when an exposure was made.  This is done by calculating the bearing between an image’s nearest points.

Obviously, this is not of the same quality as the geographic coordinates.  It’s just not a safe assumption that people always take photographs with the camera pointed in the direction they’ve been walking.  Still, this will be right some ( maybe even a lot ) of the time, and a guess is often better than no information whatsoever.  So use or ignore this at your own discretion.

Simplified Workflow

Exif Harvester is not limited to geo-tagging a batch of photos from a single GPS file. If you don’t correlate your GPS files, you can simply point to all of your GPS data. Any files with no useful data will simply not return any search hits.  You can of course point to a single file, a list of files, or to a folder full of GPS files, which will then be loaded recursively for the harvest.

Like other metadata, the geography information can be written out to XML, CSV, or other types of files.  This data is much easier to work with in Excel, SQL, or whatever tool you may prefer.

Misc

Other improvements in this release include:

• Sortable columns in the image file list
• Image previews
• A somewhat cleaner user interface

I modeled my class, Distance, on the TimeSpan struct, with a few important changes. This should make the programming model as familiar as possible. Just as a TimeSpan converts the same value ( say, 36 hours ) between different formats ( 1.5 days; 2,160 minutes, etc ), a Distance object converts between formats like feet, meters, and miles.

If this isn’t clear, a short code sample should help. The full code can be found at Distance.cs.

Distance d = new Distance();

d.Miles = 1;
Debug.Assert(d.Yards == 1760);
Debug.Assert(d.Feet == 5280);
Debug.Assert(d.Inches == 63360);

Because I’ve used the TimeSpan structure as a model, in explaining the Distance class as much as in developing it, there are a few important differences I should point out.

Enums (UnitSystem, ImperialUnits, MetricUnits)

A raw number has no meaning when divorced from its context; three what? While the elevation data in a GPS tracklog comes in meters, many US residents would prefer to see this in feet. There are three structures that provide the context that makes this data meaningful. These should be fairly self explanatory ( ImperialUnits contains inches, feet, etc while MetricUnits contains centimeters, kilometers, and more ), but it’s worth mentioning that a third struct, UnitSystem, distinguishes between the two. I felt this was a better model than simply listing every available type of 2D measurement in one enumeration ( feet, meters, and furlongs - oh my! ).

This leads to a curious paradigm for calling code - what you’ll write if you use this class. Several methods have three specific overloads, taking a decimal value and one of the following types: ImperialUnits, MetricUnits, or Enum. The third overload, taking a base Enum, lets you abstract away the specific type. The code sample below shows three ways to instantiate a Distance object with a value.

Distance d1 = new Distance(1, ImperialUnits.Mile);
Distance d2 = new Distance(2, MetricUnits.Meter);

Enum unit = userText == "mile" ? ImperialUnits.Mile : MetricUnits.Meter;
Distance d3 = new Distance(2, unit);

Normalize

While the method name may give away my background as a SQL Server developer, it has a different, but equally important meaning here. The flexibility of the Distance class would make comparisons a little bit cumbersome; one instance could describe inches while the other might be in miles or meters. Therefore, the ( static ) normalize method takes two Distance arguments, and returns the value ( as a System.Decimal ) of the second in terms of the first. In other words, if you ask for 1 yard to be normalized against 1 foot, the result will be the value 3.

Operator Overloads

I’ve overloaded + and - as well as < and > in the Distance class, which cascade so that +=, -=, <=, and >= are also redefined.

Part of the reason I chose to implement distance as a class and not a struct is to retain the object semantics most developers are used to, which includes identity comparisons by default, and the ability to change the value in an instance without creating a second instance ( eg values in a list ). This means that == is left to determine whether two instances are the same object in memory, not whether they have the same value.

Direct Access to Internal Values

Again, owing to the flexibility of representation, I felt it necessary ( although a bit dangerous ) to provide access to the underlying data.

You can set an object’s value by calling its specialized properties ( d.Miles = 2; ), however, if you have a variable representing the unit to use and would prefer to abstract the call, the SetValue method allows this ( d.SetValue(2, ImperialUnits.Mile); ).

A read-only property, InternalValue, gives developers a choice for how to perform value comparisons. This is paired with a second property, InternalUnit, of type Enum, and two others provided only for convenience, to avoid casting in client code: InternalMetricUnit, and InternalImperialUnit, plus, finally, an InternalUnitType property telling you which to use. While this may sound complicated, it is very easy to code against. You can compare the same property ( Meters, for example ) from two objects to determine whether they have the same value, avoid the mess by calling Normalize as described above, or compare the internal values and types.

System.Decimal

The decimal struct is much slower than the double, but it allows much greater precision after the decimal point. Because of the way IEEE defines floating point representations, there are many fractional values that cannot be represented exactly as a single or double precision float. Decimals, apart from using more bytes for storage, use a base-ten representation of both parts of the number, although this is done in .net code, and is truly a software solution. I feel that this is an acceptable trade-off, however; I don’t expect a great deal of math to be done against Distance objects, but I do expect people will be startled if metric calculations don’t work as expected.

Unit Tests

This class has been heavily tested, with at least one non-trivial conversion from every unit to every other unit. All code paths through Normalize have also been tested.

I’m not publishing the tests, however, as I don’t believe they have much or any use other than in preserving my sanity while I wrote this code. If I’m wrong and you could benefit from the tests, leave a comment and I’ll be happy to provide them, but, for brevity’s sake, I’ll point you at the actual code.

Because the execution plan generated by a SQL command changes for many reasons ( schema changes, data volume, etc ), knowing what a spid is doing might not be as important as knowing how the spid is doing it. For example, an ETL might suddenly run for hours, as the result of a table scan … but why? The function below can help a developer or DBA troubleshoot performance and concurrency issues.

SET ANSI_NULLS ON
SET QUOTED_IDENTIFIER ON
GO

If Exists (Select * From sys.objects Where object_id = object_id(N'dbo.QueryPlan') AND type in (N'FN', N'IF', N'TF', N'FS', N'FT'))
        Drop Function dbo.QueryPlan
GO

Create Function dbo.QueryPlan(@spid int) Returns xml As Begin
	Declare
		@plan_handle binary(32),
		@code xml

	Select @plan_handle = plan_handle
	From sys.dm_exec_requests
	Where session_id = @spid;

	Select @code = query_plan From sys.dm_exec_query_plan(@plan_handle);

	Return @code
End

This of course deserves the same warning as the input buffer function; the sys.dm_exec_requests view only tracks things that are running at the time you query it. If you’d like this function to return the most recent query plan used by a connection, change the code to look up the plan handle from sys.dm_exec_connections instead. Of course, there are good reasons for wanting a null return value for idle connections.

Because this is a scalar function, it can be used in a select list. We can extend the sample query introduced yesterday, to call dbo.QueryPlan and dbo.InputBuffer for all transactions involved in a blocking chain.

Select
	dbo.InputBuffer(spid) As RunningCode,
	dbo.QueryPlan(spid) As QueryPlan,
	*
From
	sys.sysProcesses
Where
	blocked != 0
	Or spid In (Select blocked From sys.sysProcesses)

fnbki8pjqe

Most people who use SQL Server regularly, know about the wonderful DBCC InputBuffer command. If you’re not a SQL aficionado, you can easily find out what other connections “are doing” when one of them starts causing problems. Checking the input buffer - how commands are received by the server - returns the SQL code that was most recently sent from that connection.

For example, if spid ( Server Process Identifier ) #72 is blocking other transactions, or chewing up an alarming amount of disc I/O ( especially if it’s an ad hoc query ), you can find out what it’s up to by calling:

DBCC InputBuffer (72)

Which returns something like this:

Beyond the Basics

This is a good start, but has some issues. Most importantly, the results simply aren’t available to T-SQL code. You can write an application to call DBCC externally - the Activity Monitor does this, giving you the SQL involved when you double click a process - but this limits the utility. For example, if you run sp_who2 and find a dozen questionable spids, you will have to check them individually.

On the other hand, a user-defined function ( UDF ) opens many doors. Logging comes immediately to mind, and will be the subject of a future post. A more straight-forward use for a SQL function is to simply ask what’s running on the server:

Select
	dbo.InputBuffer(spid) As RunningCode,
	*
From
	sys.sysProcesses
Where
	blocked != 0
	Or spid In (Select blocked From sys.sysProcesses)

Instead of simply getting a dump of the spid/kpid values involved, the CPU and I/O numbers, etc, this query returns at-a-glance what each connection is doing. The where clause in the query above will return only connections that are involved in a blocking chain - which is useful when debugging concurrency issues - but the concept applies no matter what you’re doing. You could rewrite the query “… Where dbo.InputBuffer(spid) Like ‘%My_Table_Name%‘ …” to see which transactions are trying to use a table, and what they’re doing with it.

The Code

The T-SQL batch below creates a function named dbo.InputBuffer to the database. This could of course be deployed to any schema; dbo is used for simplicy, as it exists by default in all databases.

SET ANSI_NULLS ON
SET QUOTED_IDENTIFIER ON
GO

If Exists (Select * From sys.objects Where object_id = object_id(N'dbo.InputBuffer') AND type in (N'FN', N'IF', N'TF', N'FS', N'FT'))
	Drop Function dbo.InputBuffer
GO

Create Function dbo.InputBuffer(@spid int) Returns varChar(max) As Begin
	Declare @code varChar(max)

	Select
		@code = SubString(
			st.text,
			(er.statement_start_offset / 2)+1,
			((Case er.statement_end_offset When -1 Then DataLength(st.text) Else er.statement_end_offset End - er.statement_start_offset) / 2) + 1
		)
	From
		sys.dm_exec_requests As er Cross Apply
		sys.dm_exec_sql_text(er.sql_handle) As st
	Where
		er.session_id = @spid;

	Return @code
End

Caveat Emptor (sys.dm_exec_requests)

There is one important difference between the code presented here, and DBCC. Checking the input buffer through the database command console gives you the last command received on any open connection. On the other hand, sys.dm_exec_requests DMV only tracks queries that are currently executing. This is a subtle distinction, but can be a very important one.

If somebody opens a query window in SSMS or Query Analyzer, runs an update query, and then doesn’t close the window once the transaction commits, DBCC will return the update query, but the function I’m presenting will not.

Alternate Version

Often, it’s useful to see NULL returned as an indicator that a given connection is dormant, but, sometimes what’s needed is the behavior from DBCC. In those cases, the code below is a better fit.

SET ANSI_NULLS ON
SET QUOTED_IDENTIFIER ON
GO

If Exists (Select * From sys.objects Where object_id = object_id(N'dbo.InputBuffer2') AND type in (N'FN', N'IF', N'TF', N'FS', N'FT'))
	Drop Function dbo.InputBuffer2
GO

Create Function dbo.InputBuffer2(@spid int) Returns varChar(max) As Begin
	Declare @code varChar(max)

	Select
		@code = st.text
	From
		sys.dm_exec_connections As ec Cross Apply
		sys.dm_exec_sql_text(ec.most_recent_sql_handle) As st
	Where
		ec.session_id = @spid;

	Return @code
End

You might instead want to use a nullable input parameter (@spid int, @exactBehavior bit = 1), or simply to deploy only whichever version of the code does what you want.

In the next episode, we’ll look at how to get the query plan for a transaction running against the server.

In the wild, this is often done with code similar to the following:

   Stopwatch sw = new Stopwatch();

   sw.Start();
   // Do some work here
   sw.Stop();

   Debug.WriteLine(sw.Elapsed.TotalMilliseconds.ToString();

This is hardly the end of the world, but it’s a bit cumbersome, especially over the course of many tests.  Perhaps a more natural construction would be:

   using (DebugStopwatch countIndexOf = new DebugStopwatch("IndexOf")) {
      // Do some work here
   }

… with the results being harvested later.  The literal text “IndexOf” can be replaced with any label that will be meaningful when you analyze the timing data.  Persistence of test results is accomplished through the magic of static classes.

While run time is almost always the most important performance metric at the end of the day, it’s often more useful to know the start and end time-stamps.  In situations when other running code can affect your results, knowing when, and not just how long, your test ran can help you make better sense of the results.  You can correlate tests to best and worst case scenarios, if you know when other, impactful things happened.

A final note:  this library commits a cardinal sin, and builds XML with a StringBuilder.  There are a few reasons for this ( performance and simplicity of the code ), and this is a very particular situation;  only numbers will ever be output, so no entities could possibly corrupt a document.

namespace ForrestCroce.Logic {
      public class BenchmarkState {
            DateTime started, ended;
            long startingWorkingSet, endingWorkingSet;
            DebugStopwatch sw;

            internal BenchmarkState(DebugStopwatch creator) {
                  startingWorkingSet = Environment.WorkingSet;
                  sw = creator;
                  started = DateTime.Now;
            }

            internal void Conclude() {
                  ended = DateTime.Now;
                  endingWorkingSet = Environment.WorkingSet;
            }

            public string XmlFragment {
                  get {
                        StringBuilder sb = new StringBuilder();
                        sb.AppendLine("\t");
                        sb.AppendLine("\t\t");
                        sb.AppendLine("\t\t\t" + started.ToString() + "");
                        sb.AppendLine("\t\t\t" + ended.ToString() + "");
                        sb.AppendLine("\t\t\t" + sw.Elapsed.TotalMilliseconds.ToString() + "");
                        sb.AppendLine("\t\t");
                        sb.AppendLine("\t\t");
                        sb.AppendLine("\t\t\t" + startingWorkingSet.ToString() + "");
                        sb.AppendLine("\t\t\t" + endingWorkingSet.ToString() + "");
                        sb.AppendLine("\t\t");
                        sb.AppendLine("\t");
                        return sb.ToString();
                  }
            }
      }

      public class DebugStopwatch : Stopwatch, IDisposable {
            BenchmarkState status;
            string label;

            public DebugStopwatch() {
                  status = new BenchmarkState(this);
                  StopwatchManager.Add(this);
                  Start();
            }

            public DebugStopwatch(string debugLabel) {
                  label = debugLabel;
                  status = new BenchmarkState(this);
                  StopwatchManager.Add(this);
                  Start();
            }

            public void Dispose() {
                  Stop();
            }

            public new void Stop() {
                  base.Stop();
                  status.Conclude();

                  string text = string.IsNullOrEmpty(label) ? DateTime.Now.ToString() : label + ": ";
                  text += (Elapsed.TotalSeconds + Elapsed.TotalMilliseconds).ToString("#,##0.000") + " ms";
                  System.Diagnostics.Debug.WriteLine(text);
            }

            public BenchmarkState State { get { return status; } }

            public string Label { get { return label; } set { label = value; } }
      }

      public static class StopwatchManager {
            static Dictionary byInstanceID = new Dictionary();

            public static DebugStopwatch Create(string label) {
                  DebugStopwatch sw = new DebugStopwatch(label);
                  return sw;
            }

            public static void Add(DebugStopwatch sw) {
                  int id = byInstanceID.Count;
                  byInstanceID.Add(id, sw.State);
            }

            public static string Xml {
                  get {
                        StringBuilder sb = new StringBuilder();
                        sb.AppendLine("");

                        foreach (KeyValuePair item in byInstanceID)
                              sb.Append(item.Value.XmlFragment);

                        sb.AppendLine("");
                        return sb.ToString();
                  }
            }
      }

}

Sometimes you need to translate between the two.  Think of absolute urls as the third normal form of the web.

The complete solution has 138 lines in two classes, but this includes fetching the content from the internet.  Filling in a host name and path for each relative link is easy. You can download the full implementation

• ResourceQualifier.cs
• WebUtility.cs

The juicy bits are below:

public string Code() {
	HtmlDocument doc = WebUtility.GetPage(originalHtml);
	RecursiveQualifier(doc.DocumentNode);

	return cleanHtml = doc.DocumentNode.OuterHtml;
}

private void RecursiveQualifier(HtmlNode node) {
	QualifyNode(node);

	foreach (HtmlNode child in node.ChildNodes)
		RecursiveQualifier(child);
}

private void QualifyNode(HtmlNode node) {
	if (node.HasAttributes)
		foreach (HtmlAttribute a in node.Attributes)
			if (string.Compare(a.Name, "src", StringComparison.OrdinalIgnoreCase) == 0 || string.Compare(a.Name, "href", StringComparison.OrdinalIgnoreCase) == 0)
				if (Uri.IsWellFormedUriString(a.Value, UriKind.RelativeOrAbsolute) && !(new Uri(a.Value, UriKind.RelativeOrAbsolute).IsAbsoluteUri))
					a.Value = QualifyUrl(a.Value).ToString();
}

public static Uri Qualify(string baseUri, string relativePath) {
	if (Uri.IsWellFormedUriString(relativePath, UriKind.Absolute))
		return new Uri(relativePath);

	if (!Uri.IsWellFormedUriString(baseUri, UriKind.Absolute))
		return null;

	Uri b = new Uri(baseUri, UriKind.RelativeOrAbsolute);

	return new Uri(b, relativePath);
}

Note:  The Qualify methods don’t agree, but download the source code, and you’ll see why.

All resources that can be called out to in html use either a src or an href attribute:  images, flash movies, style sheets, scripts, music, etc.  Xhtml 2 allows any element to be a link, so finding particular tag names doesn’t work.  Fortunately, Html Agility Pack comes to the rescue;  this gem is the XmlDocument of sloppy, malformed html.

Recursion lets you start at the root node and walk down the tree, checking every element for the attributes we’re interested in, and “fix” the links.  The Uri class takes care of what would otherwise be tedious string parsing with “/folder” and “../../parent/path” in urls.

The open source will come slowly, but do innovative and very useful things.  General ideas, findings, and snippets of code will also find their way to these pages.  Stay tuned.

C# and Microsoft SQL Server will be the general focus of this blog, although the topics will stray.