# Monday, August 11, 2003

The age old question. If you ever saw one of my TechEd sessions or WebCasts, you know that I am a big fan of using the DataReader, especially when you are doing Web DataBinding. I am all about the firehose, forward only cursor. Today I actually replaced a DataReader with a DataSet, so I think that I need to tell the world the story. J

 

So my app that runs each Sunday morning to get data over HTTP and regex out stuff was bogging down. What happens is that I have a DataReader on the client that grabs the URLs, RegEx patterns, etc from a table for the main application “loop” to process the URLs and save the data to the database. There are about 30,000 records stuffed into the datareader and the stored procedure that powers it has to do a join to the table I am adding data into on each iteration of the loop to make sure that if the process stops and restarts, I don’t reprocess any duplicate URLs.

 

So all of a sudden (this code has been in production for 15 months, and on .NET 1.1 for 4 months) I started to get timeouts when I read data from the DataReader. Randomly this would happen, maybe once ever few weeks. I never really tracked it down. So this week I would run the process and every 10th record would cause a timeout. It was a timeout when I tried to read data from the next row in the DataReader, on the 10 row. I start and stop and this happened a zillion times. I spent a few minutes playing with some settings, etc, but more of the same. Oddly enough, setting the command behavior of the command that filled the reader to SequentialAccess did not even let me read data from the first record, it returned an error saying that it can only look at data starting at the 10th row.

 

I have not discovered the problem here, but it must have something to do with the buffer, I must have been stuffing way too much data in there. So I said DUH, let’s use a DataSet. Well I never looped through a DataTable before, so here is how it goes:

 

Private void DataTableLoop (DataTable dt) {

 

   foreach (DataRow dr in dt.Rows) {

      foreach (DataColumn dc in dt.Columns) {

         Response.Write (dr[dc] +”


”);

      }

    }

}

posted on Monday, August 11, 2003 3:54:28 PM (Eastern Daylight Time, UTC-04:00)  #    Comments [4] Trackback

1:47:51

That was my triathlon time today! I am very happy. Teammate Andy Catlin got out of the water a few seconds before me (we both were at 15 minutes) and we had a hard transition (lots of rocks, and the bikes were far from the water). The bike was brutal (55 minutes). Almost all uphill, about 75% of the bike was uphill.  We had a 4 mile downhill at one point that was so super steep hill-I hit 47 miles per hour! That is almost as fast as a car… Transition from bike to run was not that bad. The run was hard, but ok, about 25-26 minutes. Andy finished about 5 minutes behind me, way to go dude…

posted on Monday, August 11, 2003 2:19:29 AM (Eastern Daylight Time, UTC-04:00)  #    Comments [0] Trackback
# Saturday, August 9, 2003

Radio Uninstalled

Radio is completely off my machine. This is good since it was slow and clunky.

This is a day of liberation for Clemens and I. We are now using our Radio subscriptions just to upstream our RSS feeds to the Userland cloud dasBlog. Very cool. So now all the folks who have not resubscribed can still see the feeds at Radio, but please please please subscribe to the new feed.

What is funny is the Clemens said he did not have enough time to document everything, but his documentation is much better than Radio's! Besides my forthcoming cache code contribution to dasBlog, I have to say that Clemens rocks the house, so buddy, I owe you a few beers when we are in KL next week speaking at Tech·Ed Malaysia.

 

posted on Saturday, August 9, 2003 11:17:06 PM (Eastern Daylight Time, UTC-04:00)  #    Comments [0] Trackback

Taper Day

 

Today I do nothing but sit at home, drink drink drink water and eat pasta. Tomorrow is the big race.

Last night I went out with some buds, including Rob Wlodarczyk who leaves for a job at the Evil Empire on Tuesday.

posted on Saturday, August 9, 2003 6:42:06 PM (Eastern Daylight Time, UTC-04:00)  #    Comments [2] Trackback
# Friday, August 8, 2003

Overcome Parkinson's Law

 

Parkinson’s Law is taking over. The basic thesis of Parkinson is: ‘Work expands so as to fill the time available for its completion.’

 

So I have called a meeting to discuss 1 small bug, should take 5 minutes, but call the meeting for 30 minutes. The meeting will last 30 minutes!

 

Further explanation from Parkinson:

 

“General recognition of this fact is shown in the proverbial phrase 'It is the busiest man who has time to spare.' Thus, an elderly lady of leisure can spend the entire day in writing and dispatching a postcard to her niece at Bognor Regis. An hour will be spent finding the postcard, another in hunting for spectacles, half an hour in a search for the address, an hour and a quarter in composition, and twenty minutes in deciding whether or not to take an umbrella when going to the pillar box in the next street. The total effort that would occupy a busy man for three minutes all told may in this fashion leave another person prostrate after a day of doubt, anxiety, and toil.”

 

So this has a lot to do with being efficient. Overcoming Parkinson’s Law is one the most liberating things that you can ever do. A big problem I have with meetings today in corporate American. They last forever and never accomplish anything. I hate when someone calls a meeting and there is no clear reason for it and then says “Stephen why don’t you start off the meeting.” Argh. So I put an end to this. Now my new rules for meetings are (and if they don’t meet this criteria, I don’t attend the meeting, unless the person calling the meeting is “important” J):

 

1.         No meeting lasts more than 45 minutes

2.         All meetings are "stand-up" meetings, nobody sits down

3.         There are no meetings without an agenda

4.         No meetings should have any food or drink allowed in them

5.         No meetings at odd hours: 5pm, obvious lunch times, etc

 

When I was CTO over at Zagat, I also instituted a meeting free Friday, something that you may want to try at your place too. I got away with it by sending an all day meeting invite to my entire staff called “Staff Meeting” that had a recurrence on every Friday and then when other people used Outlook/Exchange to schedule a meeting, they were always booked on the Friday!!!

 

Even if you can’t control the 5 items above, you can always ask the meeting organizer to circulate an agenda before the meeting, this will get them thinking more efficiently!

posted on Friday, August 8, 2003 2:32:11 PM (Eastern Daylight Time, UTC-04:00)  #    Comments [19] Trackback

Up with das Blog 1.1!

I am now up. If you are upgrading from 1.0 to 1.1, make sure you read the notes.

Time to work on the cache engine, Clemens has challenged me :)

[08:32] Peakbagger: :I am up now dude. :)
[08:32] Clemens (viel zu warm!): should work much better
[08:32] Peakbagger: looks pretty smooth
[08:33] Clemens (viel zu warm!): now... make it faster
[08:33] Clemens (viel zu warm!): ;)
[08:34] Peakbagger: that I can do

 

posted on Friday, August 8, 2003 12:51:22 PM (Eastern Daylight Time, UTC-04:00)  #    Comments [13] Trackback
# Thursday, August 7, 2003

I have to admit that I am totally hooked on the The Rozenshtein Method. My buddy, Richard Campbell showed it to me a year or two ago and I have been hooked ever since. I recently demoed it at TechED in Dallas, a recent WebCast, VSLive in New York and will be showing it off at TechEd in Malaysia next week. I have gotten lots of email and positive feedback so I decided to blog it here.

 

Here is how it works. You need a crosstab query.  You have to move rows into columns. You also need ANSI 92 SQL that will run in any database.Well there are several ways to do this, but the most generic and one of the most powerful ways is called the Rozenshtein Method, which was developed by the Russian mathematician David Rozenshtein. This technique was taken from his book: Optimizing Transact-SQL : Advanced Programming Techniques.

 

First let’s look at the desired results. We want to take the orders data from Northwind and pivot the sales date (aggregated by month) as columns with the sum of the total sales in the row grouped by customer. It would look something like this:

 

CompanyName TotalAmount Jan Feb Mar…(etc)

Company1        100              25 33   10

Company2           467                 76 62    87

(etc)

 

The TSQL query to do this is, go ahead and run it in Northwind in SQL Server:

 

SELECT  CompanyName, SUM((UnitPrice*Quantity)) As TotalAmt,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-1)))) AS Jan,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-2)))) AS Feb,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-3)))) AS Mar,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-4)))) AS Apr,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-5)))) AS May,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-6)))) AS Jun,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-7)))) AS Jul,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-8)))) AS Aug,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-9)))) AS Sep,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-10)))) AS Oct,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-11)))) AS Nov,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-12)))) AS Dec

FROM         Customers INNER JOIN

                      Orders ON Customers.CustomerID = Orders.CustomerID INNER JOIN

                      [Order Details] ON Orders.OrderID = [Order Details].OrderID

Group By Customers.CompanyName

 

So how does this work?

 

This method uses Boolean aggregates, so that each column has a numeric expression that resolves each row as a zero or one and that value (0 or 1) is multiplied by your numeric expression (Like TotalSales or (UnitPrice*Quantity). That is all there is to it, quite simple. But wait, there’s more to explain:

 

We want to create columns for each Month in our data. To find a month use DatePart. But we need to subtract the DatePart value (1-12) from the amount you’re looking for (1 for Jan, 2 for Feb, etc) as shown here for January:

DatePart(mm,OrderDate)-1

 

So that true = zero, false > 0 or < 0. For example if the month you were looking for was January and the DatePart was 1 and you subtract 1 from that value you get 0, which is true. If you are looking for March you would get -2 and that would be false.

 

Next you have to compute the sign of the expression and get the absolute value like so:

ABS(SIGN(DatePart(mm,OrderDate)-1)))

 

This will give us a positive value. Remember 0 is still true. Now subtract the value computed from 1 in order to get a 0 or 1 from the value of your expression (the Boolean aggregate). The code is:

(1-ABS(SIGN(DatePart(mm,OrderDate)-1))))

 

For example if you had March return 3 from the Datepart, 3-1=2 and 1-2 =-1. The absolute value is 1. This will always return 0 or 1. If your expression was zero, the value is now one. If was one, the value is zero.

 

Last step. Taking the SUM of the Boolean values will give you a count of the values that qualify. So you can find out how many sales you made in Jan, Feb, etc. So now multiply the value by the price and quantity, but remember its now one = true. Take a look here:

 

SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-1)))) AS Jan

 

If its zero, nothing gets added, if its one, you get the value of the sale. The sum of the total expression is the total of sales for the month. If you have a DatePart that is evaluated to 0 then ((UnitPrice*Quantity)*0) is 0 and those results are ignored in the SUM. If you have a month that matches your expression resolves to 1 and ((UnitPrice*Quantity)*1) is the value of the sale.

 

How easy!

 

But wait, there’s more! Suppose you wanted two values combined? Compute each value down to zero or one separately. Now you can use AND by multiplying, OR by adding (and reduce to 1 or 0 using SIGN).

 

Ok, have fun!!!

posted on Thursday, August 7, 2003 8:38:37 AM (Eastern Daylight Time, UTC-04:00)  #    Comments [7] Trackback
# Wednesday, August 6, 2003

The Architect

I think everyone who wanted to see this movie has seen it and it is safe to post this now. I post this in honor of Jack, Kathleen, Jay and all the others last night that told me that they were still plugged into the Matrix:

The Architect - Hello, Neo.

Neo - Who are you?

The Architect - I am the Architect. I created the matrix. I've been waiting for you. You have many questions, and although the process has altered your consciousness, you remain irrevocably human. Ergo, some of my answers you will understand, and some of them you will not. Concordantly, while your first question may be the most pertinent, you may or may not realize it is
also irrelevant.

Neo - Why am I here?

The Architect - Your life is the sum of a remainder of an unbalanced
equation inherent to the programming of the matrix. You are the
eventuality of an anomaly, which despite my sincerest efforts I have been unable
to eliminate from what is otherwise a harmony of mathematical precision.
While it remains a burden to sedulously avoid it, it is not unexpected, and
thus not beyond a measure of control. Which has led you, inexorably, here.

Neo - You haven't answered my question.

The Architect - Quite right. Interesting. That was quicker than the
others.

*The responses of the other Ones appear on the monitors: Others? What
others? How many? Answer me!'*

The Architect - The matrix is older than you know. I prefer counting
from the emergence of one integral anomaly to the emergence of the next, in
which case this is the sixth version.

*Again, the responses of the other Ones appear on the monitors: Five
versions? Three? I've been lied too. This is bullshit.*

Neo: There are only two possible explanations: either no one told me,
or no one knows.

The Architect - Precisely. As you are undoubtedly gathering, the
anomaly's systemic, creating fluctuations in even the most simplistic equations.

*Once again, the responses of the other Ones appear on the monitors:
You can't control me! Fuck you! I'm going to kill you! You can't make me
do anything!*

Neo - Choice. The problem is choice.

*The scene cuts to Trinity fighting an agent, and then back to the
Architects room*

The Architect - The first matrix I designed was quite naturally
perfect, it was a work of art, flawless, sublime. A triumph equaled only by its
monumental failure. The inevitability of its doom is as apparent to me
now as a consequence of the imperfection inherent in every human being,
thus I redesigned it based on your history to more accurately reflect the
varying grotesqueries of your nature. However, I was again frustrated by
failure. I have since come to understand that the answer eluded me because it
required a lesser mind, or perhaps a mind less bound by the parameters of
perfection. Thus, the answer was stumbled upon by another, an intuitive program,
initially created to investigate certain aspects of the human psyche.
If I am the father of the matrix, she would undoubtedly be its mother.

Neo - The Oracle.

The Architect - Please. As I was saying, she stumbled upon a solution
whereby nearly 99.9% of all test subjects accepted the program, as
long as they were given a choice, even if they were only aware of the choice
at a near unconscious level. While this answer functioned, it was obviously
fundamentally flawed, thus creating the otherwise contradictory
systemic anomaly, that if left unchecked might threaten the system itself.
Ergo, those that refused the program, while a minority, if unchecked, would
constitute an escalating probability of disaster.

Neo - This is about Zion.

The Architect - You are here because Zion is about to be destroyed.
Its every living inhabitant terminated, its entire existence eradicated.

Neo - Bullshit!

*The responses of the other Ones appear on the monitors: Bullshit!*

The Architect - Denial is the most predictable of all human responses.
But, rest assured, this will be the sixth time we have destroyed it, and we
have become exceedingly efficient at it.

*Scene cuts to Trinity fighting an agent, and then back to the
Architects
room.*

The Architect - The function of the One is now to return to the
source, allowing a temporary dissemination of the code you carry, reinserting
the prime program. After which you will be required to select from the
matrix 23 individuals, 16 female, 7 male, to rebuild Zion. Failure to comply
with this process will result in a cataclysmic system crash killing everyone
connected to the matrix, which coupled with the extermination of Zion will
ultimately result in the extinction of the entire human race.

Neo - You won't let it happen, you can't. You need human beings to
survive.

The Architect - There are levels of survival we are prepared to
accept. However, the relevant issue is whether or not you are ready to accept
the responsibility for the death of every human being in this world.

*The Architect presses a button on a pen that he is holding, and
images of people from all over the matrix appear on the monitors*

The Architect - It is interesting reading your reactions. Your five
predecessors were by design based on a similar predication, a
contingent affirmation that was meant to create a profound attachment to the rest
of your species, facilitating the function of the one. While the others
experienced this in a very general way, your experience is far more
specific. Vis-a-vis, love.

*Images of Trinity fighting the agent from Neos dream appear on the
monitors*

Neo - Trinity.

The Architect - Apropos, she entered the matrix to save your life at
the cost of her own.

Neo - No!

The Architect - Which brings us at last to the moment of truth,
wherein the fundamental flaw is ultimately expressed, and the anomaly revealed as
both beginning, and end. There are two doors. The door to your right leads
to the source, and the salvation of Zion. The door to the left leads back to
the matrix, to her, and to the end of your species. As you adequately put,
the problem is choice. But we already know what you're going to do, don't
we? Already I can see the chain reaction, the chemical precursors that
signal the onset of emotion, designed specifically to overwhelm logic, and
reason. An emotion that is already blinding you from the simple, and obvious
truth: she is going to die, and there is nothing that you can do to stop it.

*Neo walks to the door on his left*

The Architect - Humph. Hope, it is the quintessential human delusion,
simultaneously the source of your greatest strength, and your greatest
weakness.

Neo - If I were you, I would hope that we don't meet again.

The Architect - We won't.

posted on Wednesday, August 6, 2003 10:42:00 AM (Eastern Daylight Time, UTC-04:00)  #    Comments [3] Trackback
# Tuesday, August 5, 2003

Yukon is in LA...

Up to my ears in coding and writing my MSDN Magazine article on Yukon. While NDAs are still in effect, I can let you know that Microsoft has made public the notion of writing a stored procedure in a CLR .NET Language. After hacking around most of the afternoon, I am marking this day down where I got my first CLR Stored Procedure working. Stay tunded to my MSDN Magazine feature story on Yukon that will be on the stands at the PDC.

posted on Tuesday, August 5, 2003 3:56:02 PM (Eastern Daylight Time, UTC-04:00)  #    Comments [16] Trackback
# Monday, August 4, 2003

New mail has arrived...

When I got home today, my brand new custom made Torelli Brianza racing bike arrived at my door. Nohting beats hand made Italian frames. It arrived just in the nick of time since I have a major tri this weekend. Components are a Campagnolo Veloce/Mirage mix. This is a steel frame, I will have this forever. I am in a state of bliss. Now off to the bike shop tomorrow to pick out some nice clipless pedals, a cycle computer and new shoes.

posted on Monday, August 4, 2003 10:45:33 PM (Eastern Daylight Time, UTC-04:00)  #    Comments [2] Trackback

Behind Every Great Man...

Is a great Editor. Just have to send some blog love to Melody Hendricks, my editor for the last 5 or 6 years. We have worked together on over a dozen conferences and maybe 50 magazine articles. Plus she puts up with all of my crap.

Thanks MelHen for making me sound smart in print. Want to edit my blog too???

posted on Monday, August 4, 2003 12:00:52 PM (Eastern Daylight Time, UTC-04:00)  #    Comments [2] Trackback