Reclaiming my work day

Posted on May 3, 2012 by David Hogue

So this isn’t a technical post, but still applies to developers. I have had trouble managing my time and the amount of work coming in. I’m sure most people that work at a desk run into this every now and then.

Here’s four things I try to do that help:

  • I try to keep my inbox empty. See http://inboxzero.com/.
  • I try not to check my email all the time. This is hard for me.
  • I try to make a note of anything that’s nagging at the back of my mind while I’m trying to focus.
  • I drink better coffee now that I’m much better at making it. :)

They don’t make things perfect, but at least I feel like I’m in control now.

Email

I lose a lot of time to things like emails coming in, IM conversations. Just one email or IM discussion can easily eat an hour. I really try to be as helpful as I can, but sometimes I don’t have time left at the end of the day.

I’ve been pretty good about implementing Inbox Zero. I rarely have any emails left in my inbox by the end of the day and I have filters for mailing lists and CC’s. However I’d like to start processing my email in batches, so I don’t have to deal with it first thing in the day. The problem is that as soon as Outlook opens and I glance at that number, I feel compelled to take a few minutes and at least sort them. And then if there’s just one that doesn’t make sense or asks me to do something I can’t get done, it nags at me the rest of the day.

Notes

Another thing that’s helping a lot is just writing down anything I can’t get to right at the moment. Sometimes I’ll be debugging some problem and I’ll notice code that could be improved, or I get an idea that would make error logging easier, or whatever. Rather than trying to make the change right away and get sidetracked, I used to try to remember that until I got done. The problem was that I would either forget about it or I’d keep thinking about it when I needed to focus on other things.

Now I just create a one line note in Evernote. I can move it and tag it later when I have time. Or sometimes I’ll just delete it outright because it doesn’t seem like such a hot idea anymore.

Posted in Software Development | Tagged , , | Leave a comment

Plugging a ConnectionProvider in to NHibernate to connect to multiple databases

Posted on February 7, 2012 by David Hogue

I recently had a scenario where I needed to connect to a variety of databases depending on context. The databases all shared the same schema, the contents were just partitioned out based on customer.

I’d originally done this with many SessionFactories (like this post does.)

After a while I ran into memory usage issues (See this post about debugging with WinDbg.)

So that’s when I started looking for another option. What I wanted to do was use one factory and supply it with connections to the right databases.

The Provider

/// <summary>
/// Connection provider that will override the default connection string and use one that's
/// been specified for this thread.
/// </summary>
public class DbSpecificConnectionProvider : DriverConnectionProvider
{
    /// <summary>
    /// Returns a new connection using the connection string set in ThreadConnectionString.
    /// </summary>
    public override IDbConnection GetConnection()
    {
        var connection = Driver.CreateConnection();
        try
        {
            connection.ConnectionString = GetConnectionStringFromContext();
            connection.Open();
        }
        catch (DbException)
        {
            connection.Dispose();
            throw;
        }
        return connection;
    }
}

This isn’t too complex really. It inherits from the default provider of DriverConnectionProvider and overrides the GetConnection function. It then gets a connection from the driver, sets the connection string and opens the connection. If there are any errors it’ll cleanup the connection before exiting.

Additional bits

You’ll also need to edit your configuration to tell it about the new provider:

<property name="connection.provider">
  My.Namespace.DbSpecificConnectionProvider, MyAssembly
</property>

And if you are using second level caching, you might need to implement a caching provider so that the cache doesn’t get cross-contaminated.

Now this hasn’t had that much testing yet, so I’m still being a little cautious with it. But if it works, it’s a nice clean solution to my specific problem.

Posted in Software Development | Tagged | 1 Comment

Where does the time go?

Posted on January 11, 2012 by David Hogue

It seems like I don’t have any time these days. I’m getting a lot done, but is it the stuff I want to be spending my time on? And could I be more efficient about getting through it?

Between work eating up as much time as I will let it, spending time with my girl, being somewhat social, upgrading people’s PCs, trying to get through a game, and trying to read more (currently in the middle of re-reading the Hobbit!) I have no free time. Oh yeah, I should also mention that trying to write a blog will eat up a bunch of time too. :)

Yeah, I want to do a lot. Probably more than I am realistically able to. However, with some work I am able to keep a surprising number of things going. Not everything I’d ever like to do, but more than I would have thought.

Some things that have helped:

  • Task lists, calendars and notebooks. This probably deserves it’s own post. I’m using a bit of GTD (aka Getting Things Done) with Outlook, Google Calendar and Evernote. There are entire blogs dedicated to this that might be a better resource.
  • Exercise and sleep. Yes they both take time, but you’re able to better spend the rest of the time if you stay on top of them. I’m honestly still working on this. It’s not easy!
  • Getting up a bit earlier. I’m usually in bed pretty early these days, which means I sometimes get up earlier than I used to. I was never a morning person, but I do see the appeal. I’m writing this in the morning before I have to work.
  • I’m spending less time on the things that don’t interest me as much. I spend far less time on Facebook now. Less time with the Xbox even than I used to. I would like to spend some free time coding again.
  • If I still don’t have time to do something at some point I have to decide what to drop. I can always take it up later if I want, but it’s still not usually an easy decision.

So that’s how I’m getting by until I can make a 30 hour day or invent time travel.

So what do you think? Was this useful? Do you have any tips to add?

Posted in Meta | Tagged , , | Leave a comment

Some WinDbg notes for troubleshooting excessive memory usage

Posted on December 30, 2011 by David Hogue

!GCRoot

A little while ago I ran into some problems with the w3wp process gobbling up tons of memory. It ate up so much memory that after running for a few hours with high traffic, it would automatically get restarted by IIS.

WinDbg isn’t the friendliest looking program, but if you know a few commands you can get quite a ways with it.

In my case the memory was used up by how we were pushing NHibernate to do some non-standard things. These same steps could be used to troubleshoot most generic memory usage problems.

Here’s the steps I followed to get through this:

Performance Monitor

  1. I have a memory dump I made with Process Explorer on the server when it had been running for 3 hours and was about to be recycled.
  2. The Gen2 heap size was what was growing up to 1.4GB. The other heaps stayed about the same size for the life of the app. Which apparently is common when there is a memory leak. I was able to tell this with performance counters and then confirm it with windbg.
  3. The type taking up the most memory is strings, after that various types of arrays. Which also seems normal, even if there weren’t a leak.
  4. From a random sampling of strings, most look like SQL fragments, but some are HTML content. Makes sense, most of the strings in this particular app are going to be HTML or SQL.
  5. WinDbg did take a very long time to do some operations on the 3GB memory dump. Using an SSD helped, but I still had to be patient for some operations
  6. Finally it started looking like the NHibernate query cache was holding on to a bunch of data. There’s a similar issue that was discovered here.
  7. The data is rooted in a static property so it will never be garbage collected. Our code was written this way since the documentation said to only create each session factory once and store it.
  8. One NHibernate SessionFactory on its own isn’t too bad, it only took up 34MB. Where it gets problematic is when we have a bunch of those in memory.
  9. In the near term, I think this can be fixed by using some WeakReferences. Longer term I’d like to use only one or two SessionFactories.
  10. I found out that a MemoryCache would be better than WeakReferences.
    • WeakReferences go away as soon as the garbage collector runs if nobody is using them, the cached items go away when memory is tight.
    • Even longer term, I’d like to get down to just a few or even one session factory. I think reading up on sharding might be the next step here.

Links:

!DumpHeap -stat

!DumpHeap -type System.String

Useful WinDbg & SoS commands:

  • .loadby sos clr – loads SoS into windbg.
    • .load C:\SomePath\sos.dll – can be used to load another version of sos.dll that was downloaded from the server.
  • .prefer_dml 1 – makes addresses in the output clickable to get more details.
  • CTRL+Break – stops the currently running command.
  • !Help – lists all SoS commands.
  • !HeapStat – lists some stats on where memory is used.
  • !EEHeap -gc – lists start and end addresses for each heap.
  • !DumpHeap -stat [Begin] [End] – Gives stats on what types are using up memory. Begin and End come from !EEHeap -gc, I wanted only Gen2.
  • !DumpHeap -Type <Type> [Begin] [End] – Lists all instances of a type, like System.String, between Begin and End.
  • !DumpObj <Address> – Prints out details on an object.
  • !GCWhere <Address> – Prints out what heap an object is in, it’s address and size.
  • !GCRoot -nostacks <Address> – Lists objects that have a reference to the object. The object at the root of the list is why the memory has not been garbage collected. This can take quite a while.
  • !ObjSize [Address] – Lists the size of the object by adding up the size of all references. Can take a while if it needs to loop through a lot.

Details:

Here’s the output of !GCRoot 00000001805a8640 on a single string that contains a fragment of SQL. I also added some information on the total size while I was poking around.

DOMAIN(00000000001B4550):HANDLE(Pinned):73015e8:Root: 000000028010c068(System.Object[])->
00000001c0180668(System.Collections.Generic.Dictionary`2[[System.Guid, mscorlib],[NHibernate.ISessionFactory, NHibernate]])-> (1272.6 MB!)
000000020c29f578(System.Collections.Generic.Dictionary`2+Entry[[System.Guid, mscorlib],[NHibernate.ISessionFactory, NHibernate]][])-> (1272.6 MB!)
00000002402b6208(NHibernate.Impl.SessionFactoryImpl)-> (34.6 MB…)
00000002402b6628(NHibernate.Engine.Query.QueryPlanCache)->
00000002402b79c0(NHibernate.Util.SoftLimitMRUCache)->
00000002402b7a60(NHibernate.Util.LRUMap)->
00000002402b7ac0(System.Collections.Hashtable)->
00000002402b7b18(System.Collections.Hashtable+bucket[])->
00000001c08cecc8(NHibernate.Util.SequencedHashMap+Entry)->
00000001c08a6628(NHibernate.Engine.Query.HQLStringQueryPlan)->
00000001c08c9cf8(System.Object[])->
00000001c08c9c90(NHibernate.Hql.Ast.ANTLR.QueryTranslatorImpl)->
00000001c08ccc48(NHibernate.Hql.Ast.ANTLR.Tree.QueryNode)->
00000001c08caa48(NHibernate.Hql.Ast.ANTLR.Tree.FromClause)->
00000001c08cac58(NHibernate.Util.NullableDictionary`2[[System.String, mscorlib],[NHibernate.Hql.Ast.ANTLR.Tree.FromElement, NHibernate]])->
00000001c08cac80(System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[NHibernate.Hql.Ast.ANTLR.Tree.FromElement, NHibernate]])->
00000001c08cb748(System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[NHibernate.Hql.Ast.ANTLR.Tree.FromElement, NHibernate]][])->
00000001c08cb538(NHibernate.Hql.Ast.ANTLR.Tree.FromElement)->
00000001c08cb650(NHibernate.Hql.Ast.ANTLR.Tree.FromElementType)->
0000000180481ef0(NHibernate.Persister.Collection.BasicCollectionPersister)->
00000001805a83f0(NHibernate.Loader.Collection.BasicCollectionLoader)-> (34.6 MB…)
00000001805a87a0(NHibernate.SqlCommand.SqlString)-> (616 Bytes…)
00000001805a8740(System.Object[])->
00000001805a8640(System.String) (200 Bytes…)

Posted in Software Development | Tagged , , , | 1 Comment

Performance Monitor Tips & Tricks

Posted on December 6, 2011 by David Hogue

I’ve been doing a lot of digging into performance monitor this last week or so and learned some new tricks. I thought I’d write this up and share what I found in case it’s useful in the future:

  • Command line tools: logman.exe, relog.exe and typeperf.exe.
    • Logman can be used to create new logs. It has one option that the GUI doesn’t have, and that is to run the same scheduled log each day.
    • Relog can be used to convert log files from the binary .blg into .csv or SQL. It can also let you pull only specific counters or time spans out of a .blg. Another use is combining counter logs from multiple files.
    • Typeperf can be used to get raw counter data from the command line.
    • There are some commands in PowerShell too, but for adding counters logman seems much less verbose.

  • I have some productions servers logging from early morning until night and the files are only around 15MB or so.
    • They’re only logging selected counters and only logging data once a minute.
    • Some counter logs can get large very quickly because they grab a lot of data every few seconds, but the file size seems to be capped around 3GB.
    • Here is the list of counters I’m currently capturing. I’d love to hear any suggestions if there are useful ones I missed.
      \.NET CLR Exceptions(_Global_)\# of Exceps Thrown / sec
      \.NET CLR Interop(_Global_)\# of Stubs
      \.NET CLR Loading(_Global_)\Bytes in Loader Heap
      \.NET CLR Loading(_Global_)\Current Assemblies
      \.NET CLR Loading(_Global_)\Rate of Assemblies
      \.NET CLR Memory(_Global_)\# Bytes in all Heaps
      \.NET CLR Memory(_Global_)\# Induced GC
      \.NET CLR Memory(_Global_)\% Time in GC
      \.NET CLR Memory(_Global_)\Gen 0 heap size
      \.NET CLR Memory(_Global_)\Gen 1 heap size
      \.NET CLR Memory(_Global_)\Gen 2 heap size
      \.NET CLR Memory(_Global_)\Large Object Heap size
      \ASP.NET\Application Restarts
      \ASP.NET\Request Execution Time
      \ASP.NET\Request Wait Time
      \ASP.NET\Requests Current
      \ASP.NET\Worker Process Restarts
      \ASP.NET Applications(__Total__)\Compilations Total
      \ASP.NET Applications(__Total__)\Errors During Execution
      \ASP.NET Applications(__Total__)\Request Execution Time
      \ASP.NET Applications(__Total__)\Request Wait Time
      \Memory\% Committed Bytes In Use
      \Memory\Available MBytes
      \Memory\Pages/sec
      \PhysicalDisk(_Total)\Avg. Disk sec/Read
      \PhysicalDisk(_Total)\Avg. Disk sec/Write
      \Process(_Total)\Handle Count
      \Process(_Total)\Virtual Bytes
      \Processor(_Total)\% Processor Time
      \Web Service(_Total)\Bytes Received/sec
      \Web Service(_Total)\Bytes Sent/sec
      \Web Service(_Total)\Get Requests/sec
      \Web Service(_Total)\Post Requests/sec

  • There are a couple buttons in the toolbars for copying and pasting settings.
    • These can be used to paste the settings into notepad and edit it as XML.
    • It was useful for using the same settings on multiple servers as well as getting the names of counters for the command line.
    • The Windows 7 Performance Monitor has a Scale Selected Counter option in the right click menu that will automatically change the scale setting so that the graph fits in the window.

Examples

As an example, here’s a couple graphs of a server I was testing a while ago. First, traffic to the server. It doesn’t look like it was all that busy, almost 5 requests a second on average:

And here’s memory usage, you can see how the Gen 2 heap size grows and grows up to around 1.5GB when the app pool recycles:

Posted in Software Development | Tagged , , , | 1 Comment

Finally merged my old blog over to this one

Posted on October 28, 2011 by David Hogue

I swear this is the last post I’m writing for a while on the topic of blogging. Really!

It took me a while, but I finally merged the content from my old blog to this one. It wasn’t too hard since they’re both WordPress. Just export and then import. The parts that took the longest were finding a way to exclude the old posts from the RSS feed and setting up permanent redirects for all the posts and feeds.

Hopefully the transition goes smoothly from here on out. I’ll be watching my logs, stats, and webmaster tools over the next few days to make sure.

So now I need to get to writing some real content. There’s a lot of old (and probably outdated) content here now and I need to get it more up to date.

Posted in Meta | Tagged , | Leave a comment

On Working Remotely and Practicing Writing Skills

Posted on October 24, 2011 by David Hogue

I’ve been doing a decent amount of work outside the office recently (or telecommuting as they used to call it.) This has been working pretty well, but I’ve been trying to improve my communication skills since that’s been a little more difficult.

That’s a good chunk of the reason I started my original blog: to get practice writing. I figured I’d never get better without practice, and practicing in public certainly makes me more careful about what I write and how sloppy I let my writing get.

I’ve been writing emails more since I’m not getting as much face-to-face communication. I’ve noticed it’s almost more about what I don’t write than what I do write. I’ve written giant wall of text emails before, and I think that caused people to just skim it. It takes a lot of work to write a good email that will get your ideas across and get people to respond with what you want.

Posted in Technology | Tagged , , | Leave a comment

Blog Redesign

Posted on October 23, 2011 by David Hogue

I’m updating the design of this place so it doesn’t have the stock WordPress theme going on anymore. It was a decent enough look, but I wanted something that was mine.

I’m no designer and I know I could find someone more talented to do this kind of thing, but this isn’t exactly a commercial site. I’ll probably keep tweaking the look a bit to make it a little better, but I want to stick with something that looks kind of like this.

Someday soon, I’m going to try to move my old content over to this blog.

Then I’ll have to actually spend time writing some content instead of distracting myself with technical things to tweak.

Oh, and the redesign looks decent on a mobile browser. Or at least it’s pretty usable.

Posted in Meta | Tagged | Leave a comment

Stupid Router Tricks (or how to use DD-WRT to extend the range of a network)

Posted on October 21, 2011 by David Hogue

I recently found myself in a situation where I had no direct internet connection where I was staying. I did find a public wifi signal, but it was very weak.

At first I tried sitting over on the side of the room near the window and raising my laptop up higher to get a better signal. Even with that, the signal was still barely usable. It might have worked for sending an email or loading a single web page, but I needed to get some real work done.

After fighting with it for a while, it occurred to me that I had an old wifi router with me. A good old Linksys WRT54-GL.

Originally I had Tomato on it, so I connected it to my laptop with an ethernet cable and got to work. I was able to get it to find the network and connect. Now I could put the router in the top corner of the window where the signal was best and run a wire over to my laptop.

Being hardwired worked, but it still wasn’t ideal. So I got to wondering if I could simply extend the range of the network. To make a long story short, I couldn’t figure out how to do it with Tomato. There is a way, but you need to setup both routers to get it to work.

So I installed DD-WRT. It has a feature I’ve never seen on any other router: it supports virtual wireless interfaces. This meant I could have it connect to the public wifi while at the same time being another access point.

So I set it up to repeater mode, gave it everything it needed to connect to the wifi signal, and added a virtual interface:

Before I was done, I wanted to go and setup some security. It was nice I could do this separately for each interface:

After doing all this and rebooting a few times, it worked! I had a router that would essentially use a hotspot as a wan port. It was a little finicky at first, but once it started working it was rock solid.

I was still able to setup my own network, run a DHCP server, and do everything else like normal. The router would NAT everything to the hotspot as if it were my ISP. I’m assuming that the hotspot was also running it’s own NAT as well, but that didn’t matter to me.

After I got all that to work, I setup a VPN and tunneled all my traffic through it. Just in case someone malicious was also using the hotspot. It also opened up a lot of possibilities for me like forwarding a port on a public IP back to me or someday running IPv6.

Posted in Technology | Tagged , | Leave a comment

Reconfiguring the load balancer to check via HTTP

Posted on October 13, 2011 by David Hogue

So this comes from a work experience. For a project, I was looking into reconfiguring the load balancer we were using…

We’re using loadbalancer.org. Which uses Linux-HA which uses ldirectord to monitor servers.

There are a few options for how it tests to see if a server is down. We had been using connect. What that does is it just opens a TCP connection on port 80. If it opens successfully, it assumes the server is alive. The problem with that was that the app pool was stopped, but other app pools were still running so IIS was still accepting TCP connections.

What I want to do is change it to negotiate. What negotiate does is it sends an HTTP request and expects to get specific content back.

I tried negotiate last week on QA servers, and it immediately marked every server as unavailable. Not a good sign. What I finally realized is that it’s sending the HTTP request to the server’s primary IP and not the load balanced IP. As soon as I made the IIS web accept connections on that IP, it worked as advertised.

I’ve setup the web on our QA servers to accept connections on port 3333 with a specific host header. That way the load balancer can check if the app pool is running and the file can be stored in our project, but it’s unlikely someone will accidentally browse to it. (I’m thinking about reversing this decision for production honestly.)

After I made the changes to QA, I tried stopping app pools and causing crashes by changing the version of asp.net used. I never got any errors in my browser and the server was marked as non-nactive.

This could use a little more testing, but I think it’s a good solution.

Posted in Software Development | Tagged , | Leave a comment