Ruminations of idle rants and ramblings of a code monkey

Getting started with IoT

.NET Stuff | IoT | StreamInsight
So … I’ve started playing with the whole Internet-of-Things stuff that Microsoft has been talking about lately. It’s funny … I’ve been doing this kind of stuff for several years, with Microsoft’s own product and it’s finally getting traction at Microsoft. Largely, I think, because of two things: first, Windows 10 IoT edition (which is pretty cool) and second, the Azure Event Hubs. And, honestly, I can’t say that I’m just “getting started” with IoT – I’ve been doing it for a while. But I am hopping on the current wave coming out from Redmond and, overall, I like what I’m seeing. If you weren’t aware of Event Hubs, it’s a CEP service in the cloud. It scales to millions of events per second (they claim). In that, it has some of the same functionality as StreamInsight. However, getting StreamInsight to scale to millions of events per second would, admittedly, be pretty tough … that’s a lot of raw throughput. But part of how Event Hubs accomplishes this is to limit the functionality provided – the sources and sinks are strictly limited to those the team has elected to include and there’s no extensibility. It also doesn’t seem to provide the number of options that you have with StreamInsight. Not that this is a bad thing … depending on what you are trying to do, this may be absolutely fine. Also, the integration into the Azure Machine Learning with Event Hubs is pretty cool – though you can do that with a custom UDO or UDF in StreamInsight. And it’s not an either-or thing … you don’t have to use simply one or the other. In fact, there is, I think, far more usages for a combination scenario, with StreamInsight on-prem feeding to Event Hubs in the cloud. First, since there’s a lot more that you can do with StreamInsight locally (due to its extensibility), there’s quite a bit of processing you can add to the data that you are streaming, including far more advanced analytics as well as downsampling and data cleansing. Also, with local connectivity to other devices, you can respond immediately to events and conditions by issuing control commands to these other devices that are on your local network. Rather than an “Internet of Things”, it’s an “Intranet of Things”. That locally processed information can then be forwarded to the cloud Event Hub if you so desire. Think, for example, of facilities management. You have various sensors around the facility measuring all kinds of things – temperature, humidity, air quality (including CO, CO2, volatile organics, flammable gases and more) – and taking appropriate action or raising alerts. Then, this would be downsampled (using, say, a hopping window in StreamInsight) and then streamed to the cloud-based Event Hub for multiple facility monitoring and tracking. Naturally, I had to go get myself a bunch of devices. I’ve gotten an Arduino R3, Netduino 3 with WiFi and Raspberry Pi and started playing with them. My first task : duplicate the Raspberry Pi IoT Weather Station demo. Done … but took a little longer than expected and included a complete brain-fart where I hooked the 5V wire from the RasPi to the ground on the weather shield. Surprisingly, nothing appears damaged but, needless to say, it didn’t work either … and they both got quite warm. But first … now that I’ve had these for about a week, here are some thoughts. Arduino Uno R3: This is a neat little device and it’s really, really inexpensive. When it comes to embedded, build-it-yourself devices, this is the one that has, by far, the most options and most importantly (especially when you are trying to figure the thing out), the most sample code out there for using sensors and devices. It’s also open source and many of the components are as well. The controller is a 16Mhz ATmega. Yup, that’s right … only 16Mhz. The same clock speed as the first 386’s – but it’s RISC-based, so it’s a different architecture. It’s also got 32KB of flash and 2KB of SRAM … so it’s actually very limited when it comes to program size (in the flash) and working set (SRAM).  There are various flavors of the Arduino, including “clones” (for lack of a better term) and there’s also sew-on (wearable) Arduino-based systems (called LilyPad). The biggest downside to the Arduino is that the development environment really makes you appreciate Visual Studio. It’s bad … really bad. And if you leave it running too long – or maybe it’s through a sleep/resume cycle – it’ll starting chewing up CPU and memory. The programming language is C/C++-based. (Late breaking development … apparently there is a Visual Studio plug-in for Arduino. I will have to check it out). The debugger is non-existent. To debug, you’ll be writing out to the serial port and reading it. The IDE has a handy tool to monitor your serial/USB port to show the output. Still, it’s remarkably capable and powerful for the things it was designed for and the LilyPad stuff is quite intriguing. (Example: GPS embedded in a kid’s backpack, with cellular uplink to report location periodically. Kick it up a notch to record UV exposure as well). Power requirements are quite low … it’ll run just fine from a USB connection (500mA). It can be had for as little as $10. Netduino 3 WiFi: This is an Arduino-shield compatible microcontroller that runs the .Net MicroFramework – a mini version of .Net for embedded system. This latest rev of the Netduino has a 168 Mhz processor, 1408KB of flash and 164KB of RAM. As the name implies, it also as 802.11 Wi-Fi built in, including support for 802.11N – but only to the 2.4Ghz band. That – right there – makes this, for me, a far more interesting and useful device than the Netduino, especially for the journey upon which I am embarking. Yes, there are WiFi and Ethernet shields for the Arduino but this is baked-in to the board. It also has a unique (as far as I can tell) Go-Bus that allows you to quickly and easily plug in additional devices, including additional Arduino shields. The board itself will run on a USB connection but once  you start adding things to it, you’ll need to have supplemental power. Like the Arduino, it’s a truly embedded system and has no built-in support for any kind of display – though you can add Arduino-based LCDs or OLEDs for a simple display. Development is exclusively with Visual Studio using C# or and you get the full debugging experience right in Visual Studio. More powerful than the Arduino, it’s also more expensive at $70. Raspberry PI 2 Model B: Second full version of the open-source Raspberry PI. This is a full computer that’s the size of a credit card. It sports a 900Mhz quad-core ARM processor with 1 GB of RAM, has a dedicated camera and display interface, HDMI port, 4 USB 2.0 ports, microSD and Ethernet all on-board and runs various flavors of (ported) Linux including Ubuntu, OpenElec, Pidora, and a flavor just for the RasPi called Raspbian as well as Windows 10 IoT edition. Development is done with Python, Scratch, C++ or various other environments (Sonic Pi looks like it’d be really cool for my son as he’s into music) or – if you install Windows 10 IoT edition on it (as I did), you can use Visual Studio 2015. Unlike the ‘duino-based boards above, it won’t reliably run just from a USB connection (500mA for USB 2.0, 900mA for USB 3.0) so you’ll want to get a higher-powered micro USB power supply. Also, unlike both of the ‘duino boards above, it doesn’t have a built-in analog-to-digital converter (ADC) so if you want to use any analog sensors (and there are plenty of them), you’ll need to wire in an ADC. That said, of the three, this is, by far, the most powerful – but that’s not a fair comparison. RasPi is a general-purpose computer that someone with modest needs to use as their personal computer – or as a media center – or as a full web server/DNS server/DHCP Server … you get the idea. It’s also a very, very affordable $40. There are others out there as well. I’m also particularly interested in the Intel Galileo boards as they are Arduino-shield compatible but, like the RasPi, will run Windows 10 IoT edition (at least R1, right now … R2 runs Windows Embedded).     The Project: Weather Station Intra/Internet of Things Edition Over the next … well, however long but I hope to be done by the end of the summer … I’ll be building out a duplicate and extension of the Raspberry Pi IoT Weather Station demo mentioned above with the Netduino as well as extending the Rasp Pi/Win10 version. We’ll set up the weather station to stream to an Intranet-Of-Things server running StreamInsight that will then publish the data internally using a modified version of Microsoft’s Connect The Dots. This will be running StreamInsight and also publish a cleansed, down-sampled stream to the Azure Event Hub. By the end of it, I hope to have a full weather station, including wind and rain meters as well as GPS, in my back yard … and, if all goes as I hope, it’ll incorporate feeds from Weather Underground of other weather stations in my area (for comparison and alerts of incoming fronts/weather changes) as well as publish to them. I’ll also hopefully have a couple of these things scattered around the house doing things like air quality, light and presence in addition to temperature and humidity. For the Intranet-Of-Things server, I’ll probably be using my recently purchased Asus VivoMini that’s running Windows Server 2012 R2. Just today, I got all of the code running on the Netduio to access the built-in sensors of the SparkFun Weather Shield – light, temperature, humidity and pressure. There were several challenges that came out of this … and a pretty nice I2C device bus class for Netduino as well. I’ll be covering that in my next post.

Houston TechFest Sessions

Community | StreamInsight | .NET Stuff | User Groups
Houston TechFest is tomorrow at the Reliant Center. And, of course, I will be speaking. This year, I am doing two sessions: Back to Basics: .NET Essentials You May Have Forgotten (3:40 PM – 4:40 PM) Now that the .NET Framework has been out for 10 years and so many tools are available to make things easier, many developers seem to have forgotten some of the core principles of .NET. In this presentation, I'll review the fundamentals of the .NET Framework and what makes all of that magic happen. From the CLR to IL to memory management and garbage collection, we'll touch on these core concepts that every developer should know. I originally did this presentation at HDNUG and it was very well received – better than I had thought it would be. However, because the timeframe is a bit more limited at TechFest vs. HDNUG, I’ll need to pare this down a little bit so I don’t go over. Introducing StreamInsight 2.1 (4:50 PM – 5:50 PM) StreamInsight provides real-time analytics on large volumes of data with a response time measured in milliseconds, bringing a new level of capabilities to the Sql Server platform. This session will provide an overview of StreamInsight, its capabilities and use cases. It will also provide details on StreamInsight 2.1, what's been added and how it makes your real-time applications easier and more robust. I originally delivered this at Sql Saturday in Baton Rouge. I’ve tweaked and honed it a bit. Some will be familiar to my other StreamInsight intro presentations – how it works, etc. There are some updates on more use cases that we’ve seen “out in the wild” in the past year or so. StreamInsight is coming into its own now too … we are seeing more and more interest and adoption as folks get a better understanding of where it can add to their business. I hope to see you there!

StreamInsight: Simple sample of adapters and Server.Connect

.NET Stuff | StreamInsight | Code Sample
This is in response to a question on the StreamInsight forum where a developer was asking for a sample that uses Server.Connect. Before I get into the details, notes: I used the PatternDetector sample from the StreamInsight Product Team Samples on CodePlex. Rather than write something from scratch, I just converted the existing project to VS 2010 and tweaked it. Add a reference to System.ServiceModel to the project. You need to make sure that you defined an instance for StreamInsight in the setup and installed the service. And start it. Starting it helps. I had to change the creds for the StreamInsight service to use my credentials instead of Network Service because I got a funky error (that I’ve never seen before) with the Event Flow Debugger. It may have something to do with the fact that I’m running on a domain controller (don’t say it … I know … but it’s just a dev machine and I need to AD for some Hyper-V VM’s that I run). I’d recommend trying to do it with Network Service but if it doesn’t work, you’ve been warned. The error is below: Security Support Provider Interface (SSPI) authentication failed. The server may not be running in an account with identity 'DEVBIKER\J Sawyer'. If the server is running in a service account (Network Service for example), specify the account's ServicePrincipalName as the identity in the EndpointAddress for the server. If the server is running in a user account, specify the account's UserPrincipalName as the identity in the EndpointAddress for the server. You need to make sure that you copy the relevant DLL’s to the folder for the StreamInsight service host folder. By default, this is C:\Program Files\Microsoft StreamInsight 1.1\Host. These can be found in the \bin folder for the PatternDetector project and are: StreamInsight.Samples.Adapters.SimpleTextFileReader StreamInsight.Samples.Adapters.SimpleTextFileWriter StreamInsight.Samples.UserExtensions.Afa Create a folder for the input and output folders. Actually, if you are running under your own account, this won’t be necessary. If you are running under Network Service, you will. And make sure that you give Network Service appropriate permissions. The User Defined Operator in the sample that I chose caused an issue when running remotely. This surprised me … usually things run pretty much unchanged. I didn’t have the time or energy to debug that so I just changed it so that it was no longer necessary. So … the changes. In void Main, I changed the startup code to: if (inProc) using (Server server = Server.Create(streamInsightInstanceName)) { RunApp(server); } else { using (Server server = Server.Connect(new System.ServiceModel.EndpointAddress("http://localhost/StreamInsight/" + streamInsightInstanceName))) { RunApp(server); } }   As you can see, I refactored everything in the using block to a new method called “RunApp”. This made it cleaner (I thought). I suppose that I could have done it another way, but this just seemed right. In RunApp, I changed to code to create the application to check for the existence of the application and, based on that, get or create the application object. This is below: // Create application in the server. The application will serve // as a container for actual CEP objects and queries. Console.WriteLine("Creating CEP Application"); Application application; if (!server.Applications.ContainsKey(appName)) { application = server.CreateApplication(appName); } else { application = server.Applications[appName]; } And that is (typically) all that you need to do to change a StreamInsight app to run either in proc or remote. As I mentioned before, I did have to change the query and remove the reference to the UDO … which was very strange because I’ve never had issues with extensions before. I’m guessing that it had something to do with the implementation of the UDO. You can download it below Program.cs (the only changed file) for the Pattern Detector sample below:  

StreamInsight User Defined Aggregate: Standard Deviation

.NET Stuff | StreamInsight | Code Sample
Here’s a quick UDA to do standard deviation of a window. I found it interesting that I had to take the IEnumerable<double> source and call ToArray(). If I didn’t, it would throw a NullReferenceException, although why is something of a mystery. It would be nice if I could pass in the Average from the query since that’s already calculated by the StreamInsight engine but no dice. Note: I’ve not done any performance testing … it was copied from MSDN. Use at your own risk … /// <summary> /// Static class with UDA extensions for standard deviation /// </summary> public static class StandardDeviation { /// <summary> /// Extension method for the UDA. /// </summary> /// <typeparam name="T">Payload type of the stream.</typeparam> /// <param name="window">Window to be passed to the UDA</param> /// <param name="map">Mapping from the payload to a float field in the payload.</param> /// <returns>Aggregation result.</returns> [CepUserDefinedAggregate(typeof(StdDevDouble))] public static double StdDev<T>(this CepWindow<T> window, Expression<Func<T, double>> map) { throw CepUtility.DoNotCall(); } } /// <summary> /// A UDA to calculate the standard deviation of a window /// </summary> public class StdDevDouble : CepAggregate<double, double> { /// <summary> /// Computes the aggregation over a window. /// </summary> /// <param name="source">Set of events contained in the window.</param> /// <returns>Aggregation result.</returns> public override double GenerateOutput(IEnumerable<double> source) { double[] values = source.ToArray(); double mean = values.AsParallel().Average(); double standardDev = values.AsParallel().Aggregate( 0.0, // do this on each thread (subtotal, item) => subtotal + Math.Pow((item - mean), 2), // aggregate results after all threads are done. (total, thisThread) => total + thisThread, // perform standard deviation calc on the aggregated result. (finalSum) => Math.Sqrt((finalSum / (source.Count() - 1))) ); return standardDev; } } Using it in a query is simple – just make sure that you add a using statement for the namespace containing the aggregate’s definition. var stdDeviation = from e in sourceQueryStream group e by e.ItemId into eachGroup from window in eachGroup.HoppingWindow( TimeSpan.FromSeconds(10), TimeSpan.FromSeconds(1), HoppingWindowOutputPolicy.ClipToWindowEnd) select new { ItemId = eachGroup.Key, StdDev = window.StdDev(e => e.Value), Avg = window.Avg(e => e.Value), Min = window.Avg(e => e.Value), Max = window.Avg(e => e.Value), Count = window.Count() };

TF209001: NullReferenceException in TFBuild Logger with Custom Task

.NET Stuff | TFS
This has been driving me absolutely nuts for … oh … a month or so (off and on). Here’s the scenario: I’ve got a custom task that access the Team Foundation Server to do some things before the build really gets kicking; the task is called in the BeforeEndToEndIteration target, which is one of the ones that are specified as extensible. So … nothing wrong there. The custom task itself changed around the workspace settings for the build so that we could have 1 build for each branch of the code. Switching between the branches is done based on a property that is passed when kicking off the build (/p: switch). This allows one build definition for all of the branches, rather than having a separate build for each branch or manually changing the workspace mappings before kicking off the build. The task itself worked perfectly; the builds pulled and labeled the correct branch. Running the TFBuild standalone, debugging with MSBuild Sidekick as I previously detailed showed no problems and everything was fine. But, when running the build under full Team Build caused errors and the build itself was marked “Partially Succeeded”. Here’s what the build results looked like: Searching for TF209001 gave me nada. Trying to figure out what was going on, I tracked the error to the Team Build logging component (in Microsoft.TeamFoundation.Build.Server.Logger.dll) and the error message was a generic one from a try{}catch{} block in the LogProjectStartedEvent. However, I couldn’t tell much else and talking with some of my friends at MSFT didn’t yield any information either. So it bugged me. I knew that there was nothing wrong with my task … it ran just fine on its own. Even when the error happened, it did what it was supposed to do successfully and the build was actually fine. When the task was removed, everything with the logger was fine and there were no errors. But even when my task didn’t actually remap any of the workspace mappings, the logger ran into this error. It could only be some sort of mysterious interaction between the task and the TF Build logger. Frustrated, I put this on the side burner for a while and just came back to it yesterday, determined to get to the bottom of this mystery and get everything working. Looking through everything in Reflector – something every .NET developer should have in their toolbox – didn’t show anything that really jumped out at me. The only thing that I could possibly guess was that, somehow, in the call to EnsureWorkspace(), the workspace wasn’t getting created. Checking the MSBuild properties WorkspaceName and WorkspaceOwner showed, however, that these were fine and being passed along correctly. Baffled, I finally decided to fire up cordbg and get to the bottom of this. I knew that it was a (caught) NullReferenceException so I set the debugger to break on all NullReferenceExceptions. (cordbg) ca e System.NullReferenceException Next … fire up Task Manager so that I can grab the PID of the MSBuild process that Team Build used to run the build. Then kick off the build and attach to MSBuild. Finally, the debugger stopped on the exception: First chance exception generated: (0x15e37fc) <System.NullReferenceException>   _className=<null>   _exceptionMethod=<null>   _exceptionMethodString=<null>   _message=(0x15e38e4) "Object reference not set to an instance of an object."   _data=<null>   _innerException=<null>   _helpURL=<null>   _stackTrace=(0x15e3960) <System.SByte[]>   _stackTraceString=<null>   _remoteStackTraceString=<null>   _remoteStackIndex=0   _dynamicMethods=<null>   _HResult=-2147467261   _source=<null>   _xptrs=4572092   _xcode=-1073741819 Exception is called:FIRST_CHANCE Now we’re getting somewhere. Let’s see where we are. The where command will give us a stack trace. (cordbg) w Thread 0x2168 Current State:GCUnsafe spot 0)* Microsoft.TeamFoundation.Client.TeamFoundationServer::GetService +0055[native] +0014[IL] in <Unknown File Name>:<Unknown Line Number> 1)  Microsoft.TeamFoundation.Build.Server.Logger.BuildLoggerBase::get_VersionControlServer +0055[native] +0000[IL] in <Unknown File Name>:<Unknown Lin e Number> 2)  Microsoft.TeamFoundation.Build.Server.Logger.BuildLogger::EnsureWorkspace +0317[native] +0000[IL] in <Unknown File Name>:<Unknown Line Number> 3)  Microsoft.TeamFoundation.Build.Server.Logger.BuildLogger::LogProjectStartedEvent +0093[native] +0047[IL] in <Unknown File Name>:<Unknown Line Numb er> 4)  Microsoft.Build.BuildEngine.EventSource::RaiseProjectStartedEvent +0061[native] +0008[IL] in <Unknown File Name>:<Unknown Line Number> 5)  Microsoft.Build.BuildEngine.EventSource::RaiseStronglyTypedEvent +0390[native] +0131[IL] in <Unknown File Name>:<Unknown Line Number> 6)  Microsoft.Build.BuildEngine.EngineLoggingServicesInProc::ProcessPostedLoggingEvents +0213[native] +0180[IL] in <Unknown File Name>:<Unknown Line N umber> 7)  Microsoft.Build.BuildEngine.Project::Load +1608[native] +0845[IL] in <Unknown File Name>:<Unknown Line Number> 8)  Microsoft.Build.BuildEngine.SolutionWrapperProject::ScanProjectDependencies +0319[native] +0114[IL] in <Unknown File Name>:<Unknown Line Number> 9)  Microsoft.Build.BuildEngine.SolutionWrapperProject::CreateSolutionProject +0208[native] +0098[IL] in <Unknown File Name>:<Unknown Line Number> We see the entire stack from MSBuild all the way into the logger. It turns out that the exception is happening in the GetService() method of Microsoft.TeamFoundation.Client.TeamFoundationServer. The stack trace doesn’t show use the actual source lines because we don’t have a pdb for the assembly, but it does show the IL line number, which will get us to what’s going on in ILDASM. Here’s what we find there: IL_000f:  ldfld      class [System]System.ComponentModel.Design.ServiceContainer Microsoft.TeamFoundation.Client.TeamFoundationServer::m_serviceContainer   IL_0014:  ldarg.1 In Reflector, this maps to: object service = this.m_serviceContainer.GetService(serviceType); If GetService() returns null, it’s no problem; that’s handled. But if m_serviceContainer is null … that isn’t handled and will throw an exception as soon as it tries to call GetService(). Now we’ve identified where the exception is coming from. But we still don’t know why. Investigating further, m_serviceContainer is created in the constructor of the TeamFoundationServer class, which is working just fine or we wouldn’t have even gotten this far. So how does m_serviceContainer get set to null? It turns out that it’s set to null in TeamFoundationServer::Dispose(). And only there. Hmmm … how is the TeamFoundationServer object getting disposed? It turns out that I was disposing the TeamFoundationServer class. How did this affect the logger? Simple … both components were calling TeamFoundationServerFactory.GetServer() to return an instance of the TeamFoundationServer class. When I called this method, I wrapped the TeamFoundationServer class in a using{} block (it implements IDisposable) like a good little .NET doobie. A look into GetServer(), however, shows that it creates a cache (using a Dictionary) with a single instance of the TeamFoundationServer for each unique TFS URI. It’s  static and handed to all clients that request the same URI; this single instance lives the entire lifetime of the process and is shared around. (Considering how long it can take to initialize the session context, this isn’t a bad thing.) However, because of this, when I called Dispose(), it was on the same instance that the distributed logger was using! (BTW … this is not documented anywhere that I could find!) The solution – somewhat counter intuitively – is to not dispose the disposable TeamFoundationServer object when you are done with it. (Don’t make a habit of it though, OK?) Once I took out the using{} block, all was well and the build ran without any hitch at all. This is only applicable to instances that are retrieved from TeamFoundationServerProxy.GetServer(); if you create the instance using new, then you should make sure that you dispose it when you are done. This smells all kinds of wrong to me. I don’t know what the official stance would be on this, but it smells suspiciously like a bug to me. The first thing is that the TeamFoundationServer class isn’t checking to see if it is disposed before it goes about its happy, merry way, contrary to what I thought was the recommended design pattern for implementing IDisposable; this includes checking to see if the class is disposed before doing any work in a method … and throwing an ObjectDisposedException when a method is called after being disposed. If you don’t do this, then be smart enough to realize when you’ve been previous disposed (somewhere) and reinitalize when necessary (ewww … no, that’s just wrong. Don’t do that.) So there it is, the end to the mystery.

.NET Dojo: Silverlight!

.NET Stuff | Community | Events
We’ve got the next .NET Dojos scheduled - Todd Anglin will be presenting Silverlight! There will be a dojo in Austin as well as Houston. Here are the details: Overview: Microsoft officially released Silverlight 2.0 in October 2008 as a platform for .NET developers to build rich internet applications (RIAs) that run cross-browser and cross-platform. Silverlight 2.0 introduces a whole new way of developing .NET applications. It is blurring the lines between what it means to develop for the web and the desktop, enabling .NET developers to rethink how they build and distribute applications. Topics covered include: Silverlight tools, Silverlight controls, Silverlight APIs, data access, and some security. What you will learn: Todd Anglin will guide you through a combination of lecture and hands-on labs where you will learn everything you need to know to get started with Silverlight 2.0. Specifically, you’ll learn how to work with Silverlight in Visual Studio and Expression Blend, how to use Silverlight controls, how to interact with the HTML DOM, how to access and manipulate data, how to use isolated storage, and secure your Silverlight applications. We’ll go from Silverlight 101 to 301 in about 4 hours and you’ll leave with all the essential tools you need to start building real applications with Silverlight today. Prerequisites: To fully participate in this workshop, attendees will need a laptop with the following: ·        Visual Studio 2008 Professional or higher (trial is okay) with Service Pack 1 installed ·        Expression Blend 2.0 or higher (trial is okay) with Service Pack 1 installed ·        Silverlight Tools for Visual Studio 2008 SP1 installed (free) ·        Deep Zoom Composer installed (free) ·        Silverlight Toolkit December 2008 or higher (available on CodePlex) Times, dates and registration links: Austin (Microsoft Office): March 9, 2009 1:00 – 5:00 PM. Register now. Houston (Microsoft Office): March 13, 2009 1:00 PM – 5:00 PM. Register now.

Austin's First .Net Dojo! (Windows Communication Foundation)

.NET Stuff | Community
Ever since I posted about .Net Dojo, I’ve had quite a few requests to have it elsewhere. Well, after seeing Pablo’s Day of TDD and how awesome that was, I got this little notion that it might be cool to do .Net Dojo in Austin. And … also … well, this format was kinda new and I wanted to give it a trial run or two before I started doing more. So, with that said, we’ve got the very next .Net Dojo scheduled for Austin … our first one out there! It’s going to be the same as the last Houston event and, judging from the feedback that I got on the Houston Dojo, it’s gonna rock! Here are the details: When: November 3, 2008  1:00 PM – 5:00 PM Where: Microsoft Austin Office Overview: Windows Communication Foundation (WCF) is Microsoft’s strategic technology for building distributed, connected systems. Initially released with .NET 3.0, it has been significantly enhanced in .NET 3.5 to support the various web programming models including SOAP, POX, WS-*, REST and RSS/Atom syndications.  Topics covered include: SOA basics, WCF architecture, WCF bindings, hosting WCF services, WCF behaviors, WCF security and some design best practices. What you will learn: Stephen Fulcher will guide you through a combination of lecture and hands-on labs where we will learn how to implement the various web programming models with WCF. Specifically, we will cover SOA basics, WCF architecture, WCF bindings, hosting WCF services, WCF behaviors, WCF security and some design best practices. Prerequisites: To fully participate in this workshop, attendees will need an open mind and a laptop with: -  Visual Studio 2008 Professional or higher (Trial is OK) with Service Pack 1 -  Windows PowerShell 1.0 (optional, but recommended) -  Microsoft SQL Server 2005 Standard or Express -  A CD-ROM drive to load additional components and lab files. Now that you have the details, I know that you want to go here and register now. Hope to see y’all there!

Linq Performance - Part I

.NET Stuff | Linq | Performance
Well, it’s been a while since I did my initial review of some simple Linq performance tests. Since then, I’ve done a bit more testing of Linq performance and I’d like to share that. The results are enlightening, to say the least. I did this because I’ve gotten a lot of questions regarding the performance of Linq and, in particular, Linq to Sql – something that is common whenever there is a new data-oriented API. Now, let me also say that performance isn’t the only consideration … there are also considerations of functionality and ease of use, as well as the overall functionality of the API and its applicability to a wide variety of scenarios. I used the same methodology that I detailed in this previous post. Now, all of the tests were against the AdventureWorks sample database’s Person.Contact table with some 20,000 rows. Not the largest table in the world, but it’s also a good deal larger that the much-beloved Northwind database. I also decided to re-run all of the tests a second time on my home PC (rather than my laptop) as the client and one of my test servers as the database server. The specs are as follows: Client DB Server AMD Athlon 64 X2 4400+ AMD Athlon 64 X2 4200+ 4 GB RAM 2 GB RAM Vista SP1 x64 Windows Server 2008 Standard x64 Visual Studio 2008 SP1 Sql Server 2008 x64 So, with that out of the way, let’s discuss the first test. Simple Query This is a simple “SELECT * FROM Person.Contact” query … nothing special or funky. From there, as with all of the tests, I loop through the results and assign them to temporary, local variables. An overview of the tests is below: DataReaderIndex Uses a data reader and access the values using the strongly-typed GetXXX methods (i.e. GetString(int ordinal)). With this set, the ordinal is looked up using GetOrdinal before entering the loop to go over the resultset. This is my preferred method of using a DataReader. int firstName = rdr.GetOrdinal("FirstName"); int lastName = rdr.GetOrdinal("LastName"); while (rdr.Read()) { string fullName = rdr.GetString(firstName) + rdr.GetString(lastName); } rdr.Close(); DataReaderHardCodedIndex This is the same as TestDataReaderIndex with the exception that the ordinal is not looked up before entering the loop to go over the resultset but is hard-coded into the application. while (rdr.Read()) { string fullName = rdr.GetString(0) + rdr.GetString(1); } rdr.Close(); DataReaderNoIndex Again, using a reader, but not using the strongly-typed GetXXX methods. Instead, this is using the indexer property, getting the data using the column name as an object. This is how I see a lot of folks using Data Readers. while (rdr.Read()) { string fullName = (string)rdr["FirstName"] + (string)rdr["LastName"]; } rdr.Close(); LinqAnonType Uses Linq with an anonymous type var contactNames = from c in dc.Contacts select new { c.FirstName, c.LastName }; foreach (var contactName in contactNames) { string fullName = contactName.FirstName + contactName.LastName; } LinqClass_Field Again, uses Linq but this time it’s using a custom type. In this class the values are stored in public fields, rather than variables. IQueryable<AdvWorksName> contactNames = from c in dc.Contacts select new AdvWorksName() {FirstName= c.FirstName, LastName= c.LastName }; foreach (var contactName in contactNames) { string fullName = contactName.FirstName + contactName.LastName; } DataSet This final test uses an untyped dataset. We won’t be doing a variation with a strongly-typed dataset for the select because they are significantly slower than untyped datasets. Also, the remoting format for the dataset is set to binary, which will help improve the performance for the dataset, especially as we get more records. DataSet ds = new DataSet(); ds.RemotingFormat = SerializationFormat.Binary; SqlDataAdapter adp = new SqlDataAdapter(cmd); adp.Fill(ds); foreach (DataRow dr in ds.Tables[0].Rows) { string fullName = dr.Field<String>("FirstName") + dr.Field<String>("LastName"); } cnct.Close(); LinqClass_Prop This uses a custom Linq class with properties for the values. IQueryable<AdvWorksNameProps> contactNames = from c in dc.Persons select new AdvWorksNameProps() { FirstName = c.FirstName, LastName = c.LastName }; foreach (var contactName in contactNames) { string fullName = contactName.FirstName + contactName.LastName; } LinqClass_Ctor This uses the same Linq class as above but initializes the class by calling the constructor rather than binding to the properties. IQueryable<AdvWorksNameProps> contactNames = from c in dc.Persons select new AdvWorksNameProps(c.FirstName, c.LastName); foreach (var contactName in contactNames) { string fullName = contactName.FirstName + contactName.LastName; }                                           If you are wondering why the different “flavors” of Linq … it’s because, when I first started re-running these tests for the blog, I got some strange differences that I hadn’t seen before between (what is now) LinqAnonType and LinqClassField. On examination, I found that these things made a difference and wanted to get a more rounded picture of what we were looking at here … so I added a couple of tests. And the results …       Average LinqClass_Field 277.61 DataReaderIndex 283.43 DataReaderHardCodedIndex 291.17 LinqClass_Prop 310.76 DataSet 323.71 LinqAnonType 329.26 LinqClass_Ctor 370.20 DataReaderNoIndex 401.63 These results are actually quite different from what I saw when I ran the tests on a single machine … which is quite interesting and somewhat surprising to me. Linq still does very well when compared to DataReaders … depending on exactly how you implement the class. I didn’t expect that the version using the constructor would turn out to be the one that had the worst performance … and I’m not really sure what to make of that. I was surprised to see the DataSet do so well … it didn’t on previous tests, but in those cases, I also didn’t change the remoting format to binary; this does have a huge impact on the load performance, especially as the datasets get larger (XML gets pretty expensive when it starts getting big).                                                       I’ve got more tests, but due to the sheer length of this post, I’m going to post them separately.

ASP.NET Async Page Model

.NET Stuff | Performance | Web (and ASP.NET) Stuff
I just did a Code Clinic for the Second Life .NET User’s Group on using the ASP.NET async page model and it occurred to me that it’d be a good idea to do a little blog post about it as well. I’ve noticed that a lot of developers don’t know about this little feature and therefore don’t use it. It doesn’t help that the situations where this technique helps aren’t readily apparent with functional testing on the developer’s workstation or even on a separate test server. It only rears its head if you do load testing … something that few actually do (I won’t go there right now). So, let me get one thing straight from the get-go here: I’m not going to be talking about ASP.NET AJAX. No way, no how. I’m going to be talking about a technique that was in the original release of ASP.NET 2.0 and, of course, it’s still there. There are some big-time differences between the async model and AJAX. First, the async model has nothing at all to do with improving the client experience (at least not directly, though it will tend to). Second, the async model doesn’t have any client-side goo; it’s all server-side code. And finally, there is no magic control that you just drop on your page to make it work … it’s all code that you write in the code-behind page. I do want to make sure that this clear ‘cuz these days when folks see “async” in relation to web pages, they automatically think AJAX. AJAX is really a client-side technique, not server side. It does little to nothing to help your server actually scale … it can, in some cases, actually have a negative impact. This would happen when you make additional round trips with AJAX that you might not normally do without AJAX, placing additional load on the server. Now, I’m not saying that you shouldn’t use AJAX … it’s all goodness … but I just want to clarify that this isn’t AJAX. Now, you can potentially this this for AJAX requests that are being processed asynchronously from the client. Now that we have that out of the way, let me, for a moment, talk about what it is. First, it’s a really excellent way to help your site scale, especially when you have long-running, blocking requests somewhere in the site (and many sites do have at least a couple of these). Pages that take a few seconds or more to load may be good candidates. Processes like making web services calls (for example, to do credit card processing and order placement on an eCommerce site) are excellent candidates as well. Why is this such goodness? It has to do with the way ASP.NET and IIS do page processing. ASP.NET creates a pool of threads to actually do the processing of the pages and there is a finite number of threads that will be added to the pool. These processing threads are created as they are needed … so creating additional threads will incur some overhead and there is, of course, overhead involved with the threads themselves even after creation. Now, when a page is requested, a thread is assigned to the page from the pool and that thread is then tied to processing that page and that page alone … until the page is done executing. Requests that cannot be serviced at the time of the request are then queued for processing as a thread becomes available. So … it then (logically) follows that pages that take a long time and consume a processing thread for extended periods will affect the scalability of the site. More pages will wind up in the queue and will therefore take longer since they are waiting for a free thread to execute the page. Of course, once the execution starts, it’ll have no difference on the performance … it’s all in the waiting for a thread to actually process the page. The end result is that you cannot services as many simultaneous requests and users. The async page model fixes this. What happens is that the long running task is executed in the background. Once the task is kicked off, the thread processing the thread is then free to process additional requests. This results in a smaller queue and less time that a request waits to be serviced. This means more pages can actually be handled at the same time more efficiently … better scalability. You can see some test results of this on Fritz Onion’s blog. It’s pretty impressive. I’ve not done my own scalability testing on one of my test servers here, but I think, shortly, I will. Once I do, I’ll post the results here.                                                                                                                                                                               How do you do this? To get started is actually quite easy, simple in fact. You need to add a page directive to your page. This is required regardless of which method you use (there are two). ASP.NET will then implement IAsyncHttpHandler for you behind the scenes. It looks like this: <%@ Page Language="C#" AutoEventWireup="true" CodeFile="Default.aspx.cs" Inherits="_Default" Async="True" %> Simple enough, right? Let me just add a couple of things that you need to make sure you have in place. You will need to follow the .NET asynchronous pattern for this to work … a Begin method that returns IAsyncResult and an end method that takes this result. It’s typically easiest to do this with API’s that already have this implemented for you (you just return their IAsyncResult object). There’s a ton of them and they cover most of the situations where this technique helps. Now, to actually do this. Like I said, there’s two different ways to use this. The first is pretty easy to wireup and you can add multiple requests (I misstated this during the Code Clinic), but all of the async requests run one at a time, not in parallel. You simply call Page.AddOnPreRenderCompleteAsync and away you go. There are two overloads for this method, as follows: void AddOnPreRenderCompleteAsync(BeginEventHandler b, EndEventHandler e)                         void AddOnPreRenderCompleteAsync(BeginEventHandler b, EndEventHandler e, object state) The handlers look like the following: IAsyncResult BeginAsyncRequest(object sender, EventArgs e, AsyncCallback cb, object state) void EndAsyncRequest(IAsyncResult ar)             The state parameter can be used to pass any additional information/object/etc. that you would like to the begin and the end methods (it’s a member if the IAsyncResult interface), so that can be pretty handy. The code behind for such a page would look like the following: protected void Page_Load(object sender, EventArgs e) { LoadThread.Text = Thread.CurrentThread.ManagedThreadId.ToString(); AddOnPreRenderCompleteAsync(new BeginEventHandler(BeginGetMSDN), new EndEventHandler(EndAsyncOperation)); } public IAsyncResult BeginGetMSDN(object sender, EventArgs e, AsyncCallback cb, object state) { BeginThread.Text = Thread.CurrentThread.ManagedThreadId.ToString(); HttpWebRequest _request = (HttpWebRequest)WebRequest.Create(@""); return _request.BeginGetResponse(cb, _request); } void EndAsyncOperation(IAsyncResult ar) { EndThread.Text = Thread.CurrentThread.ManagedThreadId.ToString(); string text; HttpWebRequest _request = (HttpWebRequest)ar.AsyncState; using (WebResponse response = _request.EndGetResponse(ar)) { using (StreamReader reader = new StreamReader(response.GetResponseStream())) { text = reader.ReadToEnd(); } } Regex regex = new Regex("href\\s*=\\s*\"([^\"]*)\"", RegexOptions.IgnoreCase); MatchCollection matches = regex.Matches(text); StringBuilder builder = new StringBuilder(1024); foreach (Match match in matches) { builder.Append(match.Groups[1]); builder.Append("<br/>"); } Output.Text = builder.ToString(); } } If you run this (on a page with the proper controls, of course), you will notice that Page_Load and BeginGetMSDN both run on the same thread while EndAsyncOperation runs on a different thread. The other method uses a class called PageAsyncTask to register an async task with the page. Now, with this one, you can actually execute multiple tasks in parallel so, in some cases, this may actually improve the performance of an individual page. You have two constructors for this class:     public PageAsyncTask( BeginEventHandler beginHandler, EndEventHandler endHandler, EndEventHandler timeoutHandler, Object state) and public PageAsyncTask( BeginEventHandler beginHandler, EndEventHandler endHandler, EndEventHandler timeoutHandler, Object state, bool executeInParallel){}   The only difference between the two is that one little argument … ExecuteInParallel. The default for this is false, so if you want your tasks to execute in parallel, you need to use the second constructor. The delegates have identical signatures to the delegates for AddOnPreRenderComplete. The new handler timeoutHandler, is called when the operations times out and has the same signature to the end handler. So … it’s actually trivial to switch between the two (I did it to the sample listing above in about a minute.) I, personally, like this method better for two reasons. One, the cleaner handling of the timeout. That’s all goodness to me. Second, the option to have them execute in parallel. The same page as above, now using PageAsyncTask looks like to following: public partial class _Default : System.Web.UI.Page { protected void Page_Load(object sender, EventArgs e) { LoadThread.Text = Thread.CurrentThread.ManagedThreadId.ToString(); PageAsyncTask t = new PageAsyncTask( BeginGetMSDN, EndAsyncOperation, AsyncOperationTimeout, false); } public IAsyncResult BeginGetMSDN(object sender, EventArgs e, AsyncCallback cb, object state) { BeginThread.Text = Thread.CurrentThread.ManagedThreadId.ToString(); HttpWebRequest _request = (HttpWebRequest)WebRequest.Create(@""); return _request.BeginGetResponse(cb, _request); } void EndAsyncOperation(IAsyncResult ar) { EndThread.Text = Thread.CurrentThread.ManagedThreadId.ToString(); string text; HttpWebRequest _request = (HttpWebRequest)ar.AsyncState; using (WebResponse response = _request.EndGetResponse(ar)) { using (StreamReader reader = new StreamReader(response.GetResponseStream())) { text = reader.ReadToEnd(); } } Regex regex = new Regex("href\\s*=\\s*\"([^\"]*)\"", RegexOptions.IgnoreCase); MatchCollection matches = regex.Matches(text); StringBuilder builder = new StringBuilder(1024); foreach (Match match in matches) { builder.Append(match.Groups[1]); builder.Append("<br/>"); } Output.Text = builder.ToString(); } void AsyncOperationTimeout(IAsyncResult ar) { EndThread.Text = Thread.CurrentThread.ManagedThreadId.ToString(); Output.Text = "The data is not currently available. Please try again later." } } Not much difference there. We have 1 additional method for the timeout and the registration is a little different. By the way, you can pass null in for the timeout handler if you don’t care about it. I don’t recommend doing that, personally, but that’s up to you. There you have it … a quick tour through the ASP.NET asynchronous page model. It’s clean, it’s easy, it’s MUCH better than spinning up your own threads and messing with synchronization primitives (this is mucho-bad-mojo, just say NO) and it’s got some pretty significant benefits for scalability.                                                             With that, I’m outta here. Happy coding!


.NET Stuff | Events | Web (and ASP.NET) Stuff
First, what's this .Net Dojo stuff? Well, here it is ... it is a new series of events (I'm hoping monthly) that we will be having monthly to help developers hone and sharpen their .Net Kung Fu. It will be a combination of lecture and hands-on labs ... lecture to explain the concepts (of course) and hands-on to let you get your fingers dirty playing with the technology. Since it is hands-on, you will need to bring a laptop to do the labs and (possibly) install some additional prerequisites before the session (these will be listed, of course). They will be held at the Houston Microsoft office from 1:00 PM to 5:00 PM and cover a wide range of .Net topics. I'm also hoping to be able to get some of the technical leaders here in Houston to deliver some of the sessions ... so that you'll be learning from the best around. Ben Scheirman of Sogeti has volunteered to deliver the very first .Net Dojo on the ASP.NET MVC Framework. (Didn't I tell you that I wanted to get the best around?) Here are the details: Overview: ASP.NET MVC is one of the key new features that will be included in .NET Framework 3.5 Service Pack1. It provides a framework that enables you to easily implement the model-view-controller (MVC) design pattern in your web applications, helping you build loosely-coupled, pluggable components for application design, logic and display. While not a replacement for traditional ASP.NET WebForms, ASP.NET MVC does provide a compelling alternative for developing web applications and better facilitates test driven development (TDD) in web applications. This workshop will dig into what ASP.NET MVC is, what it does, and how it is different from WebForms. What you will learn: Through a combination of lecture and hands-on labs, attendees will learn how to create ASP.NET MVC applications using Visual Studio 2008 and how to work with the key components of an ASP.NET MVC application. Additionally, they will learn how to test their MVC components and applications using TDD principles. Prerequisites: To fully participate in this workshop, attendees will need the following: An open mind A working knowledge of ASP.NET, C#, HTML and CSS. A laptop with: Visual Studio 2008 Professional or higher (Trial is OK) ASP.NET MVC Preview 4 (available as a separate download. A CD-ROM drive to load additional components and lab files. Sign up here ... but hurry ... space is limited!Hope to see you there ...