Ready for Tech*ED Malaysia

Next Friday I leave for Tech·Ed Malaysia held in Kuala Lumpur. This is the last major conference (and last TechEd) of the year for me.

We have a bunch of Regional Directors speaking:

Adam Cogan (Australia)

Tim Huckaby (US-San Diego)

Clemens Vasters (Germany)

Scott Hanselman (US-Portland)

Looking forward to some geek talk and time in KL with some of the other speakers and RDs. Looking forward to catching up with my buddy Adam Cogan, haven’t seen him since Dallas, far too long. I will get Scott Hanselman to do some drinking this time, no more lame crap from you Scott. Also looking forward to hacking some das Blog with Clemens while in KL (We will party too and his girlfriend, Pat, owes me a favor!)

I just got my machine all ready to go. Here are my sessions, the Stored Procedure session is the last time I talk about The Rozenshtein Method this year (and maybe forever if they let me talk about Yukon next year):



ASP .NET DataGrid Drill Down

Track: Developer Tools & Technologies   Code: DEV203
Room: Tun Dr Ismail A   Time Slot: Tue, August 26 12:00 PM-1:15 PM
Speakers: Stephen Forte
The ASP.NET DataGrid is an incredibly powerful and timesaving server control. Compared to "classic" ASP, with just a few lines of code you can load it with data from any data source. Formatting is accomplished by setting just a few properties. Gone are the days where you had to write lots of complex code to filter, sort and page through your data. In-place editing is also a breeze. This session will show you how easy it is to use the ASP .NET DataGrid and accomplish powerful .NET grids without sacrificing any flexibility and functionality.

.NET CF Database Development with SQL Server CE 2.0 ROI

Track: Enterprise Data Management   Code: EDM207
Room: Tun Hussien Onn A   Time Slot: Wed, August 27 4:00 PM-5:15 PM
Speakers: Stephen Forte
The decision to build an embedded application has been made. In today's environment, Return on Investment (ROI) so important that it is now a main decision making factor in deciding what platform to use. Choosing the platform that your developers can leverage their existing skills and code to build something on time and on budget is more important than a cool new language or feature. We will look at a case study of a .NET CF PocketPC application build for Professional NFL Scouts using SQL Server CE 2.0. See the ROI decisions for justifying the development effort, training the developers, leveraging current ADO code and libraries and maintenance decisions along with lines of code comparison to the other embedded tools.

Efficient and Secure Data Retrieval in Your Middle Tier Using Stored Procedures and ADO. NET.

Track: Developer Tools & Technologies   Code: DEV206
Room: Tun Dr Ismail A   Time Slot: Wed, August 27 5:30 PM-6:45 PM
Speakers: Stephen Forte
Using Stored Procedures is more efficient, secure, and easier to maintain than using in-line SQL in your application's middle tier. In this session we will look how to optimize using Stored Procedures for efficient and secure data retrieval in the middle tier of your web, windows, mobile and web services applications.

Using Regular Expressions in Windows Forms and ASP .NET

Track: Developer Tools & Technologies   Code: DEV315
Room: Tun Hussien Onn A   Time Slot: Thu, August 28 10:30 AM-11:45 AM
Speakers: Stephen Forte
Regular expressions, although popular in Perl and other UNIX/C-like languages, are unintelligible to those not familiar with them. Regular Expressions provide an incredibly powerful and compact way of matching and replacing text. Once you become serious about .NET, you'll find Regular Expressions staring you square in the face. They're there and they are very important, but the .NET documentation doesn't help much in learning about Regular Expressions and how to work with them in .NET. This session will introduce you to RegEx, using RegEx as data validation and manipulation.

Strategic Inflection Point

Andy Grove, in his bestselling book, Only the Paranoid Survive, describes the nightmare moment every leader dreads--when massive change occurs and a company must, virtually overnight, adapt or fall by the wayside-- a “Strategic Inflection Point.” The arrival of the Internet was one such Strategic Inflection Point in the tech industry. I think that we are at another.

The Strategic Inflection Point is for consulting firms today and the issue is outsourcing off shore. Large companies are outsourcing almost their entire development work offshore. JPMorgan Chase just announced 50% of its IT will now be offshore (that is oh about 4,000 development jobs). I am not going to make the case for or against outsourcing, since it is happening anyway, apparently if you have a convincing argument against is, nobody is listening to it. So why fight it, it is happening and the jobs are not coming back (at least not to New York). This is why I left consulting in July 1999.

But wait, there is still money to be made in the contractor role. First let’s say that you are running a consulting shop today and are feeling the pinch of outsourcing. Let’s look at the notion that it takes 30% of the development effort to design the application, 30% to code, 30% to test and 10% to physically implement.

I always say that the strategic advantage you bring to the table in the 30/30/30/10 universe is the design. That is the MOST important phase of any project-period.  (Don’t make me go Steve McConnell on you!) What if you set up shop to do the 30% design and 10% physically implement?  You can make the case to do the design locally and then work with the outsourcers offshore to code and test. Set up a strategic partnership with several off shore players. Or better yet set up a JV. You can then take on a project management, relationship management and architecture guidance role. I have worked with overseas outsourcing and they tend to follow specs to the letter, so if your potential customer has made the decision to outsource overseas, they will be more inclined to do the design work very seriously.

Just an idea…

Mount Rainier called a threat

What were we thinking?

http://www.msnbc.com/news/950829.asp

The age old question. If you ever saw one of my TechEd sessions or WebCasts, you know that I am a big fan of using the DataReader, especially when you are doing Web DataBinding. I am all about the firehose, forward only cursor. Today I actually replaced a DataReader with a DataSet, so I think that I need to tell the world the story. J

So my app that runs each Sunday morning to get data over HTTP and regex out stuff was bogging down. What happens is that I have a DataReader on the client that grabs the URLs, RegEx patterns, etc from a table for the main application “loop” to process the URLs and save the data to the database. There are about 30,000 records stuffed into the datareader and the stored procedure that powers it has to do a join to the table I am adding data into on each iteration of the loop to make sure that if the process stops and restarts, I don’t reprocess any duplicate URLs.

So all of a sudden (this code has been in production for 15 months, and on .NET 1.1 for 4 months) I started to get timeouts when I read data from the DataReader. Randomly this would happen, maybe once ever few weeks. I never really tracked it down. So this week I would run the process and every 10^th record would cause a timeout. It was a timeout when I tried to read data from the next row in the DataReader, on the 10 row. I start and stop and this happened a zillion times. I spent a few minutes playing with some settings, etc, but more of the same. Oddly enough, setting the command behavior of the command that filled the reader to SequentialAccess did not even let me read data from the first record, it returned an error saying that it can only look at data starting at the 10^th row.

I have not discovered the problem here, but it must have something to do with the buffer, I must have been stuffing way too much data in there. So I said DUH, let’s use a DataSet. Well I never looped through a DataTable before, so here is how it goes:

Private void DataTableLoop (DataTable dt) {

   foreach (DataRow dr in dt.Rows) {

      foreach (DataColumn dc in dt.Columns) {

         Response.Write (dr[dc] +”

      }

    }

}

1:47:51

That was my triathlon time today! I am very happy. Teammate Andy Catlin got out of the water a few seconds before me (we both were at 15 minutes) and we had a hard transition (lots of rocks, and the bikes were far from the water). The bike was brutal (55 minutes). Almost all uphill, about 75% of the bike was uphill.  We had a 4 mile downhill at one point that was so super steep hill-I hit 47 miles per hour! That is almost as fast as a car… Transition from bike to run was not that bad. The run was hard, but ok, about 25-26 minutes. Andy finished about 5 minutes behind me, way to go dude…

Radio Uninstalled

Radio is completely off my machine. This is good since it was slow and clunky.

This is a day of liberation for Clemens and I. We are now using our Radio subscriptions just to upstream our RSS feeds to the Userland cloud dasBlog. Very cool. So now all the folks who have not resubscribed can still see the feeds at Radio, but please please please subscribe to the new feed.

What is funny is the Clemens said he did not have enough time to document everything, but his documentation is much better than Radio's! Besides my forthcoming cache code contribution to dasBlog, I have to say that Clemens rocks the house, so buddy, I owe you a few beers when we are in KL next week speaking at Tech·Ed Malaysia.

Taper Day

Today I do nothing but sit at home, drink drink drink water and eat pasta. Tomorrow is the big race.

Last night I went out with some buds, including Rob Wlodarczyk who leaves for a job at the Evil Empire on Tuesday.

Overcome Parkinson's Law

Parkinson’s Law is taking over. The basic thesis of Parkinson is: ‘Work expands so as to fill the time available for its completion.’

So I have called a meeting to discuss 1 small bug, should take 5 minutes, but call the meeting for 30 minutes. The meeting will last 30 minutes!

Further explanation from Parkinson:

“General recognition of this fact is shown in the proverbial phrase 'It is the busiest man who has time to spare.' Thus, an elderly lady of leisure can spend the entire day in writing and dispatching a postcard to her niece at Bognor Regis. An hour will be spent finding the postcard, another in hunting for spectacles, half an hour in a search for the address, an hour and a quarter in composition, and twenty minutes in deciding whether or not to take an umbrella when going to the pillar box in the next street. The total effort that would occupy a busy man for three minutes all told may in this fashion leave another person prostrate after a day of doubt, anxiety, and toil.”

So this has a lot to do with being efficient. Overcoming Parkinson’s Law is one the most liberating things that you can ever do. A big problem I have with meetings today in corporate American. They last forever and never accomplish anything. I hate when someone calls a meeting and there is no clear reason for it and then says “Stephen why don’t you start off the meeting.” Argh. So I put an end to this. Now my new rules for meetings are (and if they don’t meet this criteria, I don’t attend the meeting, unless the person calling the meeting is “important” J):

1.         No meeting lasts more than 45 minutes

2.         All meetings are "stand-up" meetings, nobody sits down

3.         There are no meetings without an agenda

4.         No meetings should have any food or drink allowed in them

5.         No meetings at odd hours: 5pm, obvious lunch times, etc

When I was CTO over at Zagat, I also instituted a meeting free Friday, something that you may want to try at your place too. I got away with it by sending an all day meeting invite to my entire staff called “Staff Meeting” that had a recurrence on every Friday and then when other people used Outlook/Exchange to schedule a meeting, they were always booked on the Friday!!!

Even if you can’t control the 5 items above, you can always ask the meeting organizer to circulate an agenda before the meeting, this will get them thinking more efficiently!

Up with das Blog 1.1!

I am now up. If you are upgrading from 1.0 to 1.1, make sure you read the notes.

Time to work on the cache engine, Clemens has challenged me :)

[08:32] Peakbagger: :I am up now dude. :)
[08:32] Clemens (viel zu warm!): should work much better
[08:32] Peakbagger: looks pretty smooth
[08:33] Clemens (viel zu warm!): now... make it faster
[08:33] Clemens (viel zu warm!): ;)
[08:34] Peakbagger: that I can do

I have to admit that I am totally hooked on the The Rozenshtein Method. My buddy, Richard Campbell showed it to me a year or two ago and I have been hooked ever since. I recently demoed it at TechED in Dallas, a recent WebCast, VSLive in New York and will be showing it off at TechEd in Malaysia next week. I have gotten lots of email and positive feedback so I decided to blog it here.

Here is how it works. You need a crosstab query.  You have to move rows into columns. You also need ANSI 92 SQL that will run in any database.Well there are several ways to do this, but the most generic and one of the most powerful ways is called the Rozenshtein Method, which was developed by the Russian mathematician David Rozenshtein. This technique was taken from his book: Optimizing Transact-SQL : Advanced Programming Techniques.

First let’s look at the desired results. We want to take the orders data from Northwind and pivot the sales date (aggregated by month) as columns with the sum of the total sales in the row grouped by customer. It would look something like this:

CompanyName TotalAmount Jan Feb Mar…(etc)

Company1        100              25 33   10

Company2           467                 76 62    87

(etc)

The TSQL query to do this is, go ahead and run it in Northwind in SQL Server:

SELECT  CompanyName, SUM((UnitPrice*Quantity)) As TotalAmt,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-1)))) AS Jan,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-2)))) AS Feb,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-3)))) AS Mar,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-4)))) AS Apr,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-5)))) AS May,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-6)))) AS Jun,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-7)))) AS Jul,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-8)))) AS Aug,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-9)))) AS Sep,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-10)))) AS Oct,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-11)))) AS Nov,

  SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-12)))) AS Dec

FROM         Customers INNER JOIN

                      Orders ON Customers.CustomerID = Orders.CustomerID INNER JOIN

                      [Order Details] ON Orders.OrderID = [Order Details].OrderID

Group By Customers.CompanyName

So how does this work?

This method uses Boolean aggregates, so that each column has a numeric expression that resolves each row as a zero or one and that value (0 or 1) is multiplied by your numeric expression (Like TotalSales or (UnitPrice*Quantity). That is all there is to it, quite simple. But wait, there’s more to explain:

We want to create columns for each Month in our data. To find a month use DatePart. But we need to subtract the DatePart value (1-12) from the amount you’re looking for (1 for Jan, 2 for Feb, etc) as shown here for January:

DatePart(mm,OrderDate)-1

So that true = zero, false > 0 or < 0. For example if the month you were looking for was January and the DatePart was 1 and you subtract 1 from that value you get 0, which is true. If you are looking for March you would get -2 and that would be false.

Next you have to compute the sign of the expression and get the absolute value like so:

ABS(SIGN(DatePart(mm,OrderDate)-1)))

This will give us a positive value. Remember 0 is still true. Now subtract the value computed from 1 in order to get a 0 or 1 from the value of your expression (the Boolean aggregate). The code is:

(1-ABS(SIGN(DatePart(mm,OrderDate)-1))))

For example if you had March return 3 from the Datepart, 3-1=2 and 1-2 =-1. The absolute value is 1. This will always return 0 or 1. If your expression was zero, the value is now one. If was one, the value is zero.

Last step. Taking the SUM of the Boolean values will give you a count of the values that qualify. So you can find out how many sales you made in Jan, Feb, etc. So now multiply the value by the price and quantity, but remember its now one = true. Take a look here:

SUM((UnitPrice*Quantity)*(1-ABS(SIGN(DatePart(mm,OrderDate)-1)))) AS Jan

If its zero, nothing gets added, if its one, you get the value of the sale. The sum of the total expression is the total of sales for the month. If you have a DatePart that is evaluated to 0 then ((UnitPrice*Quantity)*0) is 0 and those results are ignored in the SUM. If you have a month that matches your expression resolves to 1 and ((UnitPrice*Quantity)*1) is the value of the sale.

How easy!

But wait, there’s more! Suppose you wanted two values combined? Compute each value down to zero or one separately. Now you can use AND by multiplying, OR by adding (and reduce to 1 or 0 using SIGN).

Ok, have fun!!!

The Architect

I think everyone who wanted to see this movie has seen it and it is safe to post this now. I post this in honor of Jack, Kathleen, Jay and all the others last night that told me that they were still plugged into the Matrix:

The Architect - Hello, Neo.

Neo - Who are you?

The Architect - I am the Architect. I created the matrix. I've been waiting for you. You have many questions, and although the process has altered your consciousness, you remain irrevocably human. Ergo, some of my answers you will understand, and some of them you will not. Concordantly, while your first question may be the most pertinent, you may or may not realize it is
also irrelevant.

Neo - Why am I here?

The Architect - Your life is the sum of a remainder of an unbalanced
equation inherent to the programming of the matrix. You are the
eventuality of an anomaly, which despite my sincerest efforts I have been unable
to eliminate from what is otherwise a harmony of mathematical precision.
While it remains a burden to sedulously avoid it, it is not unexpected, and
thus not beyond a measure of control. Which has led you, inexorably, here.

Neo - You haven't answered my question.

The Architect - Quite right. Interesting. That was quicker than the
others.

*The responses of the other Ones appear on the monitors: Others? What
others? How many? Answer me!'*

The Architect - The matrix is older than you know. I prefer counting
from the emergence of one integral anomaly to the emergence of the next, in
which case this is the sixth version.

*Again, the responses of the other Ones appear on the monitors: Five
versions? Three? I've been lied too. This is bullshit.*

Neo: There are only two possible explanations: either no one told me,
or no one knows.

The Architect - Precisely. As you are undoubtedly gathering, the
anomaly's systemic, creating fluctuations in even the most simplistic equations.

*Once again, the responses of the other Ones appear on the monitors:
You can't control me! Fuck you! I'm going to kill you! You can't make me
do anything!*

Neo - Choice. The problem is choice.

*The scene cuts to Trinity fighting an agent, and then back to the
Architects room*

The Architect - The first matrix I designed was quite naturally
perfect, it was a work of art, flawless, sublime. A triumph equaled only by its
monumental failure. The inevitability of its doom is as apparent to me
now as a consequence of the imperfection inherent in every human being,
thus I redesigned it based on your history to more accurately reflect the
varying grotesqueries of your nature. However, I was again frustrated by
failure. I have since come to understand that the answer eluded me because it
required a lesser mind, or perhaps a mind less bound by the parameters of
perfection. Thus, the answer was stumbled upon by another, an intuitive program,
initially created to investigate certain aspects of the human psyche.
If I am the father of the matrix, she would undoubtedly be its mother.

Neo - The Oracle.

The Architect - Please. As I was saying, she stumbled upon a solution
whereby nearly 99.9% of all test subjects accepted the program, as
long as they were given a choice, even if they were only aware of the choice
at a near unconscious level. While this answer functioned, it was obviously
fundamentally flawed, thus creating the otherwise contradictory
systemic anomaly, that if left unchecked might threaten the system itself.
Ergo, those that refused the program, while a minority, if unchecked, would
constitute an escalating probability of disaster.

Neo - This is about Zion.

The Architect - You are here because Zion is about to be destroyed.
Its every living inhabitant terminated, its entire existence eradicated.

Neo - Bullshit!

*The responses of the other Ones appear on the monitors: Bullshit!*

The Architect - Denial is the most predictable of all human responses.
But, rest assured, this will be the sixth time we have destroyed it, and we
have become exceedingly efficient at it.

*Scene cuts to Trinity fighting an agent, and then back to the
Architects
room.*

The Architect - The function of the One is now to return to the
source, allowing a temporary dissemination of the code you carry, reinserting
the prime program. After which you will be required to select from the
matrix 23 individuals, 16 female, 7 male, to rebuild Zion. Failure to comply
with this process will result in a cataclysmic system crash killing everyone
connected to the matrix, which coupled with the extermination of Zion will
ultimately result in the extinction of the entire human race.

Neo - You won't let it happen, you can't. You need human beings to
survive.

The Architect - There are levels of survival we are prepared to
accept. However, the relevant issue is whether or not you are ready to accept
the responsibility for the death of every human being in this world.

*The Architect presses a button on a pen that he is holding, and
images of people from all over the matrix appear on the monitors*

The Architect - It is interesting reading your reactions. Your five
predecessors were by design based on a similar predication, a
contingent affirmation that was meant to create a profound attachment to the rest
of your species, facilitating the function of the one. While the others
experienced this in a very general way, your experience is far more
specific. Vis-a-vis, love.

*Images of Trinity fighting the agent from Neos dream appear on the
monitors*

Neo - Trinity.

The Architect - Apropos, she entered the matrix to save your life at
the cost of her own.

Neo - No!

The Architect - Which brings us at last to the moment of truth,
wherein the fundamental flaw is ultimately expressed, and the anomaly revealed as
both beginning, and end. There are two doors. The door to your right leads
to the source, and the salvation of Zion. The door to the left leads back to
the matrix, to her, and to the end of your species. As you adequately put,
the problem is choice. But we already know what you're going to do, don't
we? Already I can see the chain reaction, the chemical precursors that
signal the onset of emotion, designed specifically to overwhelm logic, and
reason. An emotion that is already blinding you from the simple, and obvious
truth: she is going to die, and there is nothing that you can do to stop it.

*Neo walks to the door on his left*

The Architect - Humph. Hope, it is the quintessential human delusion,
simultaneously the source of your greatest strength, and your greatest
weakness.

Neo - If I were you, I would hope that we don't meet again.

The Architect - We won't.