I’m not sure if the crash took out IT resources (such as servers) for the IRS. I couldn’t help but think, “If the crash took out IRS servers, were they able to recover in a timely fashion”? Yeah I know it’s geeky and nerdy to be thinking about these things but I’m an IT pro—I can’t help but ponder these things!

Every IT resource—infrastructure and systems alike—should have a disaster/recovery plan. Within government agencies, they like to call this Continuity of Operations Planning or COOP. COOP is pretty comprehensive and includes not just how to recover IT but also how management succession, crisis procedures, etc. If you are part of IT and you get detailed to participate in COOP planning, most of your contributions to this COOP plan will revolve around recovering IT assets such as servers, applications, databases, and the like,

I can’t remember how many times I’ve seen network and DB administrators claim “yeah we have backups” and when disaster comes, the backups were no good and couldn’t be restored! What good is that??! What the hell is that?? See, it’s not enough that you are backing up data and applications; you must also rehearse recovery procedures using the capture back-ups so that you can confidently report to your management, “Yes, we have backups and can recover in the event of a disaster.”

In the SharePoint world, it’s not enough that you are backing up content and configuration databases—you should rehearse recovery procedures from time to time. How do you know you can recover a toasted SharePoint farm configuration if you’ve never rehearsed it and seen with your own two eyes that your backups are good? You must test your recovery procedures.

How often should you backup and test recovery procedures? Well, one can probably write a dedicated Web site just on the topic of disaster-recovery (just google it and there are tons). So, without having to discuss this too much, as a general guide I like to follow, the more critical your systems, the more backups and test-recovery procedures you should do. For less-critical apps, your frequency doesn’t have to be as extensive (the important thing is you do it). Just as an example, for critical apps—daily full back-ups with incremental back-ups during the day and test recovery procedures every 2 weeks. The “least frequent” backup schedule I’ve ever done for a “non-critical” app is bi-weekly full backups and test recoveries every 3 months.

The point is, you must backup and be ready to recover when disaster strikes. Nine years ago, the terrorists attacked WTC and the Pentagon. Last week, some lunatic had a personal grudge against the government. Who knows, maybe this week, some admin at your office spills some latte on your production server and toasts mission-critical apps. Whatever the disaster may be, be ready.

Here’s what happened. Apparently, due to security policies set forth by the organization, all Oracle connections must time-out (Oracle session status=”SNIPE”) if the connection has exceeded the “Idle Time” specification in the Oracle Profile. Without getting deep into Oracle, the server will kill the connections after the Idle Time limit. I actually had our Oracle DBA helped me diagnose the problem. The Oracle DBA set traces and we monitored the Oracle sessions. That’s when I saw that even if initially, the application will keep several connections open (connection pooling is turned on for the OracleConnection object and you will see INACTIVE status in the Oracle sessions–inactive is okay–it just wasn’t executing commands at that time), after several minutes (15 minutes is the set Idle Time), the Oracle sessions would disappear. In a few rare occassions, we actually saw the Oracle sessions for the app time-out. The sessions appear with a “SNIPE” status for a little bit and then the server eventually cleans them up.

Is this behavior reproducable? Absolutely! Create an Oracle Profile (get your DBA if you don’t have rights to do this) and set the Idle Time to a short interval like 5 minutes. Next, associate the Oracle user-name that the application will use to the profile that has the 5-minute Idle Time limit. Run the app and you should be able to connect to Oracle. Wait like 10 minutes; your next call will generate the Oracle errors. Another way to reproduce this is the run the app for the first time and then watch the Oracle sessions. Kill the sessions created by the app. You will get “ORA-00028: your session has been killed” instead of the “idle time exceeded error”. But the following Oracle commands after that error will now generate the “not logged on” error. The ODP.NET pool manager gave the application a connection that should have been discarded because it’s already dead!

The ODP.NET provider will keep threads in the pool even though the server has already killed them! What’s the fix? The fix is a parameter that you need to add to your connection string. On my Oracle connection string, I only specify Data Source, User Name, and Password; no other parameters. I added an additional parameter called Validate Connection and you have to set it equal to True. I googled all day to find what this does. You can learn more out about this at the Oracle site. When you say Validate Connection=True on the connnection string, the ODP.NET connection pool will test the connection first before it returns the connection to the requesting app. The connection string look something like:

<add key="connectionString"
     value="Data Source=ORCL;
            User ID=me;
            Password=myPassword;
            Validate Connection=True;" />

 By putting Validate Connection=True on the connection string, connectionObject.Open() will get you a connection that is either:

• Valid open connection from the connection pool
• A fresh new connection the connection pool

You’ll stop getting crappy, dead connections from the pool. What’s downside? The pool manager will now ping the Oracle server using the connection first (to test if it’s still valid) before it returns to the application. A small price to pay I say to make the application stable.

I’ve tested using the Validation Connection=True method and it works. I can kill sessions in Oracle and the app will just re-open valid connections. The Oracle instance can be shutdown and restarted–when it restarts, the pool will simply create new database connections. Another possible approach is to set the Connection Lifetime parameter to a time value that is below the Idle Time limit in the server. In theory, this method will make the ODP.NET connection pool will discard the connections before the server kills it. My only problem with this latter strategy is that this doesn’t take into account Oracle server “bounce”, firewall disconnects, physical disconnects and the like. Just go with the Validate Connection parameter–it is still application pooling; just a bit busier.

NOTE: the stuff I wrote above is applicable to ODP.NET only. The System.Data.OracleClient namespace also has OracleConnection, OracleCommand, and adapter and reader objects as well. But they’re not the same. The System.Data.OracleClient, by default, does not use application pool. I tested this several times, again, by monitoring the Oracle sessions. If you put Pooling=True in your connection string, then the application will use connection pooling. What happens when the sessions are killed or sniped and you are using System.Data.OracleClient.OracleConnection in you app? First, you get “ORA-00028: your session has been killed” just like in ODP.NET. The difference between System.Data.OracleClient and ODP.NET OracleConnection is that in the former, it’s smart enough to not keep giving the cut-off connection back to application. You’ll get ORA-00028 error once and the succeeding calls should be fine. ODP.NET on the other hand, will keep that bad connection in the pool even though it’s been bad for a while; only when the ODP.NET connection has exceeded it’s TTL that it is removed from the pool.