Court of Lord Zoltan v2

Story: A mystery of three

2011-08-05T23:00:00.000+01:00

I wrote this story a good few years ago now. I've been meaning to put it up for a while now, and only now just got the wherewithall to do so.

A mystery of three

By Andras Zoltan

"Christ", thought Andras, as he sifted through the thousands of papers on the desk "Somewhere here are the words that'll elevate me to the next level, I need to find them, bring them together to make a collage of feelings and ideas; a catastrophe of logic and reason. But where do I start?"

The bits of paper all had dark, angular scrawls all over them. On not one of them did any discernable sentence appear. Instead they seemed to have been leaked onto the paper both in the act of writing them as well as dreaming them up, as though a product of a deep trance. As soon as they caught his gaze, the words seemed to fly at him, each one's impact growing from the last; their chaos seeping into his mind as it tried to make sense of what he was seeing.

He stared at one of the pages for a few seconds, his eyes glistening with intense concentration on the babble - using each word like a pin on a map; it was impossible to remember each of the words he was seeing, he'd thought earlier, so instead he would give them entry to his unconscious, by simply being seen. This way, the word would be stored but in a way such that only its trace remained; its meaning blurred by the randomness of the words before and after, a shadow, threatening to resurface at any time.

All at once, with one of the pages in his hands, he got to his feet and started to shout the words; his voice would carry them to the heavens, their influence would be felt everywhere and by everyone. The reverberation of his voice seemed to magnify their impact on his mind, with each one a small change occurred inside him. Now he could feel them. Now he could take the power of these words and bring it together, focus it, use it. With a new purpose he started pacing, tossing the page aside and picking up a new one, he began once again to wreak language's havoc upon the world through his voice: "Elevation!!!" he cried, the room responding with a breath to acknowledge the signal being received. "Liiiiiiiightning!!!!!!!", the images of huge sparks of white appearing around him; inside his head crashes of thunder echoed, reminding him of all the times he'd seen the sky turn into an electric playing field, smelled the sweet odour of clean air after a rainstorm and been petrified by the rumbling voice of the sky as a child. He paused, almost willing for there to be a crash of thunder in the distance, knowing that such an event would petrify him for a most different reason.

Continuing on, he felt certain that right here, right now, something was happening to him, it must be embraced and encouraged. "These words have finally given one of their secrets up to me, it's now only time that decides when they come together, and me that decides exactly how they come together" he said to himself as he prepared for the next incantation. It left his mouth in all directions like a bullet from a gun, bending the walls and shaking the windows, the room wasn't built for would-be opera singers.

"HAAAUUUUNNNNTTTTT!!!!"

A door crashed far off down the corridor. Andras froze, the petrified feeling that he half-expected just before now visited upon him tenfold as a million nasty thoughts went through his mind as to what caused that noise?

Footsteps could be heard, approaching the other side of the door from the unknown; the being's appearance changing constantly inside Andras' mind as it advanced, his fear and anxiety increasing with each new form. He wondered if another echo from the depths of madness would cause the footsteps to go away, could a word be found to release him from this terrible nightmare? Is this the power of the words manifesting itself? In a last ditch effort he reached for the nearest page in his grasp, as he maneuvered it into view for him to incur it's wrath upon the demon outside, he filled his lungs with air and readied his body. His eyes scanned the letters, the syllables slotted into the empty spaces of his mind, all the strength inside him disappeared: "defeat?" he murmured, "but by what?"

The footsteps stopped, the doorknob started turning slowly, scraping its way round to let the evil on the other side in. Suddenly the catch clicked free and as the door opened a hooded figure entered the room; not the six-legged hairy demon or chainsaw maniac he was expecting, but a figure he recognised. It was him.

"Hello, Andras", the other Andras said to him.

"Well, right about the last thing I expected to happen was for another me to appear", he said in a tone of disbelief. He dropped the page and stood motionless, staring at the image before him. With the hood still up, the face seemed far more angular and foreboding than he imagined his own face to be; and his eyes, they seemed to see straight through him - a gaze like that of a heroin addict.

I stood, purposefully intimidating, and stared at him; his body frozen by fear and flummoxed by my realisation, it was amusing to see a man who'd been saturating his brain with words temporarily lost for them.

"What did you expect to happen? A genie to appear? All the world's problems solved in one fell swoop? Come on, Andras! The only reason I am here is because you summoned me here, you need me for something, to make sense of something for you."

"That's ridiculous! In any case, it still doesn't explain how you can be standing in front of me. And… and why the hell should you come here, now, when all around me is nothing but madness?" His thought patterns were clearly messed up, reeling from the barrage of the chaos of vocabulary he'd inflicted on them before.

"I should have thought that was self-explanatory." A wry smile appeared on the other Andras' face. He started walking around the room, looking down at the floor, the desk and the pages strewn all over them. The glazed look disappeared from his eyes and he moved with a certain confidence and easiness, as if he knew everything around him as having once been home.

"Madness is not something I entertain, I was merely conducting an analytical approach to the problem I'm faced with; how that's led to your appearance I don't know. It seems strange to me that you've appeared in a guise that I would never consider, let alone hallucinate." The perplexed Andras straightened up, desperately trying to elude some sense of confidence in his situation.

"Don't talk rubbish, " the hooded man snapped straight back at him, "you felt something when those words charged out of you. Release… No, a satisfaction, at being able to create images with them, of old memories resurfacing and washing over you again. I could feel it in you. There may have been an element of the scientist in you, but you're not going to convince me you didn't start to let go juuuussst a little." Very briefly a friendliness appeared in his manner, he knew more about me than I was willing to give up to anybody and he was taking pleasure in that. I watched him survey the pages once again; it was uncanny the way he seemed to observe them in order, but an order that was discernable to no other than himself.

Nothing about what I saw puzzled me. I remembered the intense cascade of feelings that gave birth to them, the hate, the pleasure, the confusion, and how they had all combined to produce a string of impressions on the paper; hurried along as if there were already millions upon millions queuing up to do the same. Nothing came out in order, a complete mish mash; and here was this fool trying to make sense of it all, though I was certainly not going to let him in on the secret just yet. I didn't even know whether there was something to make sense of, although I guess if you look at anything long enough, you'll find something in there with meaning. He was even trying to make sense of me, like he did of everything else that confronted him.

"What are you doing with all this stuff?" The second Andras muttered, turning once again to face the first one.

"I... errr, it's kinda difficult to explain, but somewhere here are the words that will release me, but I have no way of knowing which of the words I have to use or in what order. Do I have to make a sentence? Read all the words in turn? Create a play? A poem... who knows?" I considered the possibility that this explanation was unnecessary, that the other Andras already knew what I was doing, but the drive to justify my ridiculous behaviour was impossible to fight.

"So, your first urge is to read each of the words randomly and then start shouting them out like a madman!? Do you have any idea what could happen to you if somebody found out? They'd have you locked and drugged up fast, you'd only be able to remember the word 'Daadaa' by the time they'd finished with you!"

"I couldn't think of anything else, it seemed like the logical thing to do. In any case it wasn't really a decision but it, sort of, just happened. I swear something inside me has been changed or switched on after doing that."

"Damn right, you’re standing here talking to me! I think that constitutes a big change." he tapped his head through his hood thrice with his finger, making a deep hollow click with this tongue each time, insinuating insanity. Then he started towards his opposite. "Y'know the universe is supposed to explode if we touch…"

"B-but you're not made of antimatter, if you were, then you wouldn’t be able to touch anything in the room" Andras' heart started racing, he could feel his mind starting to crack, things were not right about all this, he stepped back, not sure whether to take this imposing shadow, himself, seriously - a position he'd never expected to be in.

"Ah yes, that's true, but you could be. Maybe you're the imposter in this scene, the irregularity, the blip." he accentuated this last word with amazing precision, I felt condemned.

"Think about it: I am at home here, you are not; I am in acceptance of my situation, you are not. You clearly cannot make head nor tail of what you see before you..."

"And you're saying you can? There's nothing to make sense of here! The only person who can shed any light on all this is the mad bastard that vomited it all up, and he's disappeared from existence! There's no trace of him, except what's here."

"So take a step back from the situation for the moment, isn't that what you're supposed to do? Who do you think that mystical person could be?"

I stared at him, and he at me, throughout a moment of eerie silence, as if time had stopped everywhere in the universe just to allow me to catch up with my situation. I knew the answer, I just didn't want to say it.

"You've got it, it's me. What you see in front of you is me, embodied in every word. I can't remember how long it took or why each of the words appeared, but it was me that brought them into existence."

"You can't have written all this stuff, it's impossible... Why would you create a roomful of paper covered in random words? Why would I be placed here to sift through it all? How can a hallucination have his own private storeroom of crap? It doesn't make sense!"

"You're here for your own reasons now, as I was when I wrote all this. If you feel that you're job is to make sense of what you see before you then go ahead and try. Just bear in mind that it was not sense and order that produced it, but chaos. In a way, there is a lot of you here, too, it's just that you've not realised it yet."

"How can that be? I am not you, " another portion of Andras' consciousness started to crack, "I could not have produced any of what lies here, it's alien to me. You're alien to me. It is I that must find the combination that will release me, but it is you that produced it." Why would I put myself in such a predicament anyway? he asked himself, seating himself at the table and putting his hands on his head. Nothing could explain what was going on, not even the all-knowing, all-seeing phantasm before him. The more he thought about it, there was no reason for his drive to search through the randomness that surrounded him, it was just automatic - something that had to be done.

"We are not the same person. Christ, if that were true then the universe would explode if we touched, but we are of the same essence. It must be obvious to you that we share something in common. I mean, come on! Hasn't my appearance demonstrated something to you? Doesn't my very appearance demonstrate something to you? Has it ever occured to you that the name Andras can represent two different identities, both intertwined with the same purpose?"

"But what purpose? What is about you and me that is the same? What is it that holds the key to my release from this ever-increasing nightmare? You seem to be here only to confuse things to a greater degree, to encourage that which already lives in this room." I could not function, the sanity ebbing away from me. There I was, seated at a table covered with anonymous, yet supposedly familiar, oozed-out feelings and thoughts; I felt like screaming but could select no words to describe what was inside. The hooded-me produced an empty page from his coat and drew his finger across it's surface, on it the word bled, like a punchline delivered by a mute.

Disbelief, followed by intense curiosity, appeared on Andras' face as my finger worked it's way up and down the page. I could sense him desperately trying to figure out whether or not what had appeared before him referred to him or to me. The drive for me to manufacture it was undefeatable, and there was no conscious thought given to it's inception, yet still there were ways of relating it back to our situation that he would no doubt find. Once finished I stood staring at it, held out for us both to see. It needed to be felt, even if it wasn't real.

Everything settled in the room for a moment, the two Andrasses transfixed in time as the word joined the others; making the room more complete.

"Defeat?" I asked him, dumbfounded at it's appearance yet again. He didn't reply, just stared back at me. "Am I defeated? Am I going to fail in my task to fathom your craziness? Or am I defeating you right now?" Nothing, no clue signalled by his gaze. The exasperation inside me was becoming unbearable, I felt I was being withheld something, a secret that I should know if was to continue on my travels.

"Of course you're going to fail in your task to fathom these words," he started abruptly, snapping out of his trance. He started pacing around the floor, bending slowly to let the page from his grasp. "You're going to fail in trying to make sense of me, these words, this room and anything else you turn your head to. You need me to understand this, I have to be here for anything to make sense."

I was being preached to and I knew it, dissected and repaired by this other me. I knew nothing about him, yet he knew me; perhaps he really is a figment of my imagination, I thought. If he is, he's an annoyingly mysterious one. "Without me being here you will never find the release you search for, your constantly trying to find a pattern or a solution to everything will not help you."

It was like listening to a record impressed with the ramblings of an insane sage, the words buzzing through the air in search of a listener. Still, what he said made sense. All my efforts to make sense of the situation had failed and my coup de graçe had resulted in the scaring of my life. Even now, as we talked, I was still uneasily wary of him.

"So, if you're purpose is the same as mine, and my purpose is to find the words that can release me, then..." no chance of finishing the sentence, as the hood rushed forward to interject.
"Is your purpose to find words? What makes you so sure of that? Have you been given any instructions as to what to do in your time here? You've assumed a role from the moment you got here, much in the same way that I have."

"I don't know any different to what I started doing back there."

"Just as I don't know any different to what I'm doing now."

"But, even though I feel driven to, you say my purpose here is not to find order?"

"Perhaps it's to create the madness, the chaos, much like you think mine is? How do you think all the other pieces of paper have got here?"

A bomb dropped somewhere in the distance, splintering everything within range into tiny pieces.

"This... means... that you are here purely because I am; there's something inside of me that needs to get out and you're the key to it's... it's release." The walls began to pulsate, mimicking the blood pumping through my veins as the realisation hit me. I released my head from my hands' grasp and looked, once again, at my double.

"Not just me, but the pages, too. The products of our previous visits here, and memories that have built up over the years; I haven't dreamed all of these words up on my own, y'know, just a lot of them. You're not likely to remember anything about that, though." The lack of seriousness in his voice hid the scale of what he was saying. I was standing in my (or should it be "our"?) own private storeroom of memories and ideas, represented by words on bits of paper, to be sifted through and added to at will and I didn't recognise any of it. What a ridiculous place to get stuck in with another me, I thought.

"I don't want to create chaos, though, I have no desire to eek out any more madness than the cornucopia that already adorns this room. So why am I here?".

"You think this is a one-way relationship? I may be the voice of your meanderings but I have a voice that needs to be heard too. To do this, I need a catalyst: You." He hoisted himself onto the table and sat hunched forward, staring sombrely at the opposite wall, upon which a page hung with the word "fountain" clearly etched into it; the hood chuckled shortly and turned to Andras. "It's just that, without you, my voice can emit nothing but total chaos, nothing worth saying. You inject the order into it through your presence and interaction with me."

"But you seem to able to speak properly, I mean, there's no chaos in what you're saying to me right now."

"That's as maybe, but consider what I'm like when I'm on my own. All this stuff bouncing around and nobody to express it to; I am limited just to this." His hand swooped around the room, indicating the pages. "This place is horrible on your own, let me tell you, the words around you choke the mind and confuse the soul - concentrating on them is a sure way of going mad, the only way to fit in is to continue to add more, expand the zoo as it were. With you here, though, the room and its contents are given a context, a place in time, the words and any subsequent additions take on a whole new meaning."

"But where is the product of our interaction? From what you're telling me, us coming together is supposed to lead to something being created, I see nothing; apart from the random words you continue to write as normal."

"The product is not for you, or I, to see. Only to create."

As he said this he drew another page from his coat and began writing, again with his finger. He could not explain what was drawing the energy out of him, the lines appearing as if they were already there but needed his touch to make them visible. The other Andras stood up to see what was being written this time. What word would come of this? What was inside the hood that required everything to stand still in order for it to appear?

Silence.

It was broken, "Of all the words...", Andras started, and before he could finish he was no longer there - any trace of his being purged from the room.

"Of all the words, indeed." The other Andras said, and disappeared into the air to be become nothing but a whisper…

The page sank to find its place on top of all the others. For a moment it glowed there, in the middle of the floor, its scrawled companion shouting its presence to all the others:

"Release".

Asp.Net 2 and 4 default application pool generates CS0016 IIS7.5

2011-02-11T22:26:00.001Z

Before I start – if you’ve found a bunch of other articles about this around the net, tried the fixes that are mentioned and still not getting any joy – then read on – you might find this solves your problem.

Earlier today I discovered that when I run any ASP.Net 2 or 4 application through IIS7.5 using the default application pools (which use ApplicationPoolIdentity) on Windows 2008 R2 x64 I get an error message similar to this:

Server Error in '/MvcApplication31' Application.

Compilation Error

Description: An error occurred during the compilation of a resource required to service this request. Please review the following specific error details and modify your source code appropriately.
Compiler Error Message: CS0016: Could not write to output file 'c:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\mvcapplication31\222b4fe6\4e80a86\App_global.asax.clb4bsnc.dll' -- 'The directory name is invalid. '
Source Error:

[No relevant source lines]

Source File: Line: 0

(When running an ASP.Net 2 app, the version folder is of course different)

Now, I only just picked this up today (despite doing web dev on this machine build for the last 18 months) because early on I changed the Asp.Net 2 ‘DefaultAppPool’ to use Network Service as I needed the server to make web requests to web services (PayPal etc) through a corporate proxy that uses AD authentication.

Today, however, I started a project that will be using Asp.Net 4 for, and so I was using that app pool for the first time (and hadn’t touched it all since installing .Net 4.0, VS2010 etc).

I tried uninstalling .Net completely, reinstalling it, (re)installing Asp.Net 2 & 4 but no matter what, this always kept happening.

However, with either app pool set to use Network Service it all worked fine.

Now, using my google-fu I found a few links (http://support.microsoft.com/default.aspx?scid=kb;en-us;825791, http://dudesdoingdotnet.blogspot.com/2007/11/compiler-error-cs0016-could-not-write_2486.html and http://wiki.evident.nl/Default.aspx?Page=Compiler%20error%20CS0016%20The%20directory%20name%20is%20invalid), most of which suggest that the problem is to do with %TMP% or %TEMP% folder access for Network Service, or that the folder isn't present.

Well, clearly that wasn’t the case here as these run as the dynamic IIS APPPOOL\[app pool name] built in accounts that are created for the app pool, and not Network Service. That said, I realised it's possible that the dynamic account didn't have access (or the group it belongs too), so I granted it full permissions on my %TMP%folder (it and %TEMP% both equal to C:\windows\temp) and tried again.

Still no joy.

Now the strange thing was that the folder had some pdbs and compiler traces in it – just no DLLs. So clearly files were getting created and the identity was able to write to the folder – but what was so special about the DLLs?

I tend to swear quite a lot in the office to be honest, even when something not very bad happens; but after about the 5th or 6th time of failing to fix it I was raging.

So I came home, had some dinner and decided to jump on again. This time I loaded up Process Monitor, set the filter to w3wp.exe only and watched as I generated the error again. After more filtering I found entries for the file reported in the error, with the result ‘NAME NOT FOUND’ – but noticed that other files that were also successfully created also had this, so that couldn’t be it.

Then it dawned on me that the process to watch was not w3wp – but csc.exe (if using c#). So I changed the filter and found loads of entries that referenced c:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp, with a status of ‘ACCESS DENIED’ – the user listed as IIS APPPOOL\Asp.Net v4 (for .Net 4 – it would be IIS APPPOOL\DefaultAppPool for .Net 2 on a default install of Windows Server 2008 R2).

So I gave the IIS_IUSRS groups full control of that folder – IISRESET for safety, and hit the site again.

Yay! It worked!

So, in some cases, it’s not the root temp folder C:\Windows\Temp that you need to be looking at – but this other one that belongs to the NetworkService user profile. The question is: why was this not already properly configured on my machine or, equally, is this actually the correct folder for the built in Asp.Net accounts?

My colleague’s machine didn’t have the same problem at all – so I have no idea why my machine (and I do look after it!) did. Either way at least it’s now fixed!

ObsoleteAttribute and the Expression Compiler

2011-01-28T09:23:00.001Z

I was going to post this as a question over at SO, but as I got to the end I realised a blog would probably be better (apart from the fact that only five people will read it!).

The project I'm working uses runtime objects that are produced with configuration-driven delegates built using Expression trees.

I'm doing a big functionality merge at the moment and have reached a class that, because of another that has grown, is now obsolete. So I want to make it 'properly' obsolete (generate compiler errors), go through the static codebase and change all the references.

The problem is that the class library is used, through Expressions, by a few other apps so I can't yet get rid of the class completely - I'll perhaps look at farming that job off to somebody else.

So I thought I'd just check whether the expression compiler honours the ObsoleteAttribute when marked as an error, and it doesn't:

[TestClass]
 public class UnitTest1
 {
     [Obsolete("obsolete", true)]
     public class ObsoleteClass
     {
 
     }
 
     [TestMethod]
     public void TestObsoletedClassViaExpression()
     {
         Type obsoleteType = typeof(UnitTest1).GetNestedType("ObsoleteClass");
 
         Func<object> f = 
             Expression.Lambda<Func<object>>(
                 Expression.New(obsoleteType)
             ).Compile();
 
         var o = f();
 
         Assert.AreEqual(obsoleteType, o.GetType());
     }
 }

In my case this is actually an advantage as it means the existing projects will carry on working with the updated codebase since the only reference to this type will be through dynamically generated code.

Of course once I finally remove the type those apps will no longer work - but I won't do that until they've all been updated. That said, it would be a lot easier to go through these apps an augment them to the replacement type if the Expression compiler *did* honour the ObsoleteAttribute - just follow the runtime errors (! - sorry I mean run the tests of course ;) ).

I understand, and have exploited, the fact that the Expression compiler is allowed to do things that aren't normal - skip visibility checks etc. I also understand that attributes such as ObsoleteAttribute require a compiler to honour them as, at the IL level, they are meaningless.

However, given Extression trees' ubiquity and growing importance, shouldn't its compiler honour (or have the option to honour) the ObsoleteAttribute? On the DLR, for example, this means that a language designer must code in reflection at every step to look for Obsolete(true) declarations if he/she wants to have this functionality in their language.

Like I say, it’s a tricky one because in this particular example it’s going to work in my favour – I’m just not sure it should.

Colin McRae Dirt 2 – Choppy sound and crashes in multiplayer

2010-11-03T15:05:00.001Z

I like my driving games; always have done since the original Mario Kart on the SNES way back when.

So the other day I bought Dirt 2 on Steam as I needed some driving action (my previous driving squeeze was Grid – and I can highly recommend that too). Very cool, especially with XBox 360 wireless controllers.

After clocking up 20 hours or so on the solo missions I decided to give the online game a go. Whilst in the trailer I navigated to the Multiplayer ‘board’ and hit ‘online’.

The instant I hit the button on the pad, the sound started to crackle & stutter; then I lost everything except the menu sound effects (only later for them to return for about 10 seconds). I’ve had this before (during the videos on Streetfighter 4, for example) and I just thought ‘well if the game still plays, I’ll put up with it for now’.

Only it didn’t play – not exactly anyway. The first race went okay (apart from getting battered by the opposition!), and then I went back to the menu ready for the next race to begin. When the next race began to load, the game just froze.

So I fired it up again and had another go. Got through the first race again, only on the return to the menu, it got stuck when the camera zooms out from the ‘photo’ that’s taken at the end.

All the while, the sound’s doing crazy things. Oh and the framerate in-race was not as good as it should have been either (of course it can drop slightly in Multiplayer given the extra work going on to do the networking).

Now my machine isn’t top-end (at least not until the year’s bonus comes in ;) ) but it’s still pretty beefy:

Intel Q9650 on an NVidia 780i Ultra Quad Sli board (can’t remember the manufacturer any more, possibly ASUS – but it cost a bit).
4Gb Corsair Ram as 2x2Gbs using stock timings
Two BFG GTX 280s (not OCd) in SLI
SB x-Fi Pci-ex card.

So there’s really no excuse there for the kind of crap that was going down!

I did a google search for the problem and lots of people have been reporting similar problems, but seemingly not to any decent conclusion. Lots of talk of onboard hardware, out of date drivers etc. One common thread seemed to be Win7 x64 (which I run), but ultimately my machine couldn’t be any more up-to-date driver-wise (I’m a habitual upgrader).

Lots of people were moaning about the Rapture sound system that the game uses as well – well, mine was setup at it’s defaults (and still is).

So I gave up.

A couple of days later I was mucking about on the Games for Windows (GfW) screens in-game and stumbled across the Voice settings. Guess what – as soon as the UI for that settings panel was displayed, the exact-same sound interference kicked in again! I exited that screen and it went away straight away too.

Now, I don’t use a headset with a microphone (don’t play any games where verbally abusing everyone else will either help or add anything); so what I’m about to say will NOT help those of you that do.

I tried disabling the voice stuff in GfW, but it didn’t make any difference.

So instead I pressed ALT+ENTER to get out of full screen and opened the Windows Recording Devices UI from the Volume Control. I went through and disabled all the recording devices.

Went back to the game window and back into fullscreen (ALT+ENTER again) and all of a sudden the sound was clean again on the Voice UI in the GfW overlay.

So, immediately I went to the multiplayer menu again – hit ‘Online’ as I was before and this time – no crackle, no stuttering or anything: everything working exactly as it should.

I then went on to play (and lose!) a bunch of online races, no pauses between loading, frame-rate nice and crisp as in Single Player and, best of all, sound stayed intact all the way through.

So – there you have it – disable the recording devices (probably only have to disable one or two of them, but I don’t do any audio recording on my machine anyway, so I’m happy with them all switched off) and have your problem fixed!

There’s clearly an underlying bug here – possibly at the Dirt 2 level, or equally at the GfW level, so that needs to be fixed; although the game is nearly a year old now so I doubt it ever will be.

In the meantime I can now get back to getting my ass handed to me by supafly teenagers with too much time on their hands; like I used to be 15 years ago.

Map System.TimeSpan to xs:duration for the DataContractSerializer

2010-09-21T15:18:00.001+01:00

I’m working on a new RESTful service (I’m using Asp.Net MVC for this) and I’m keen to make it as easy to integrate with as across as many client platforms as possible.

I’m going to be using XML exclusively, as I it enables me to produce schemas that our service clients will be able to download and then see the shape of each object that the service operations will expect.

I also want to employ XML schema validation on the data coming in from the request. Doing this will trap most data errors it gets further down the request pipeline, thus protecting my code; but will also ensure that the caller gets nice verbose error messages – XML Schema Validation is pretty explicit on what’s gone wrong!

Thus, I’ve wired up an MVC action to produce a schema for the relevant objects using the XsdDataContractExporter class. I will simply point users at this, alongside documentation for each of the operations; which will include schema type names from that ‘live’ schema.

While testing out one of the types, I noticed that a TimeSpan member was being serialized as xs:string and not as xs:duration. I consulted an MSDN topic I should now know by heart (given how many times I’ve looked at it) to check the support for TimeSpan in the DataContractSerializer and sure enough it’s there; but I couldn’t understand why, if DateTime is indeed mapped to the XML DateTime type, it’s not mapped to the XML Duration type.

So I’ve written a TimeSpan type called XmlDuration that is implicitly convertible to System.TimeSpan but which, when you expose it as a member on a Data Contract, presents itself as xs:duration. This then means that if you enable schema validation on your incoming XML, the input string will be validated against the rules attached to the XML Duration type, instead of being simply a string that allows any content.

The code is as follows. There’s one really long line in there which is a Regex that I’ve split into multiple string additions purely for this post; you can join the strings back up again if you so desire:

 /// <summary>
/// This type is a TimeSpan in the .Net world but, when 
/// serialized as Xml it behaves like an XML duration type.  
/// For more about the duration data type in XML - see
/// http://www.w3.org/TR/xmlschema-2/#duration
/// 
/// You should use this on types that intend to send a 
/// timespan to ensure clients can read the data in a 
/// conformant manner.
/// 
/// Note that when the type writes out to XML, it starts 
/// with days; not years.  That is because .Net timespan 
/// only expresses days.
/// 
/// If the original duration string is preserved from 
/// the input XML, then the same duration instance will 
/// serialize out using that same string.
/// </summary>
[XmlSchemaProvider("GetTypeSchema")]
public sealed class XmlDuration : IXmlSerializable
{
    /// <summary>
    /// When this instance is loaded from XML, this is 
    /// the original string.
    /// </summary>
    public string ValueString { get; private set; }
 
    /// <summary>
    /// The inner .Net TimeSpan value for this instance
    /// </summary>
    public TimeSpan Value { get; private set; }
 
    public XmlDuration() { }
 
    public XmlDuration(TimeSpan duration)
    {
        Value = duration;
    }
 
    public XmlDuration(XmlDuration duration)
    {
        Value = duration.Value;
        ValueString = duration.ValueString;
    }
 
    public XmlDuration(string duration)
    {
        try
        {
            Value = TimeSpanFromDurationString(duration);
            ValueString = duration;
        }
        catch (ArgumentException aex)
        {
            throw new ArgumentException
                ("Invalid duration (see inner exception", "duration", 
                aex);
        }
    }
 
    public static implicit operator XmlDuration(TimeSpan source)
    {
        return new XmlDuration(source);
    }
 
    public static implicit operator TimeSpan(XmlDuration source)
    {
        return source.Value;
    }
 
    private static Regex RxRead = new Regex(@"^(?<ISNEGATIVE>-)?" +
        @"P((?<YEARS>[0-9]+)Y)?((?<MONTHS>([0-9])+)M)?((?<DAYS>([0-9])+)D)?" +
        @"(T((?<HOURS>([0-9])+)H)?((?<MINUTES>([0-9])+)M)?((?<SECONDS>([0-9]" + 
        @")+(\.[0-9]{1,3})?)S)?)?$", RegexOptions.Compiled);
 
    /// <summary>
    /// Constructs a new TimeSpan instance from the pass XML 
    /// duration string (see summary on this type for a link 
    /// that describes the format).
    /// 
    /// Note that if the input string is not a valid XML 
    /// duration, an argument exception will occur.
    /// </summary>
    /// <param name="value"></param>
    /// <returns></returns>
    public static TimeSpan TimeSpanFromDurationString(string value)
    {
        TimeSpan toReturn = TimeSpan.MinValue;
        var match = RxRead.Match(value);
 
        match.ThrowIf(m => m.Success == false, "value", 
            "The string {0} is not a valid XML duration".FormatWith(value));
 
        bool isNegative = false;
        int years = 0, months = 0, days = 0, hours = 0, minutes = 0;
        double seconds = 0;
 
        var group = match.Groups["ISNEGATIVE"];
        isNegative = group.Success;
 
        group = match.Groups["YEARS"];
        if (group.Success)
            years = int.Parse(group.Value);
        group = match.Groups["MONTHS"];
        if (group.Success)
            months = int.Parse(group.Value);
        group = match.Groups["DAYS"];
        if (group.Success)
            days = int.Parse(group.Value);
        group = match.Groups["HOURS"];
        if (group.Success)
            hours = int.Parse(group.Value);
        group = match.Groups["MINUTES"];
        if (group.Success)
            minutes = int.Parse(group.Value);
        group = match.Groups["SECONDS"];
        if (group.Success)
            seconds = double.Parse(group.Value);
 
        //now have to split the seconds into whole and fractional.
        //note - there is clearly a potential for a loss of fidelity
        //here given that we're expanding years and months to 365 and
        //30 days respectively. There's no perfect solution - although
        //you can simply ask your web service clients to express all
        //durations in terms of days, hours, minutes and seconds.
        int wholeSeconds = (int)seconds;
        seconds -= wholeSeconds;
        toReturn = new TimeSpan((years * 365) + (months * 30) + days,
            hours, minutes, wholeSeconds, (int)(seconds * 1000));
        if (isNegative)
            toReturn = toReturn.Negate();
 
        return toReturn;
    }
 
    #region IXmlSerializable Members
 
    /// <summary>
    /// Returns a qualified name of 
    /// http://www.w3.org/2001/XMLSchema:duration
    /// </summary>
    /// <param name="xs"></param>
    /// <returns></returns>
    public static XmlQualifiedName GetTypeSchema(XmlSchemaSet xs)
    {
        return new XmlQualifiedName
            ("duration", "http://www.w3.org/2001/XMLSchema");
    }
 
    public System.Xml.Schema.XmlSchema GetSchema()
    {
        //see the static GetTypeSchema method.
        return null;
    }
 
    public void ReadXml(System.Xml.XmlReader reader)
    {
        string s = reader.ReadElementContentAsString();
        if (s.IsNotWhitespaceOrNull())
        {
            Value = TimeSpanFromDurationString(s);
            ValueString = s;
        }
        else
            Value = TimeSpan.MinValue;
    }
 
    public void WriteXml(System.Xml.XmlWriter writer)
    {
        StringBuilder sb = new StringBuilder();
 
        //if we have the original duration string then we write that back out.
        if (ValueString.IsNotWhitespaceOrNull())
            writer.WriteValue(ValueString);
        else
        {
            if (Value.Ticks < 0)
                sb.Append('-');
 
            bool isFractionalSeconds = 
                ((double)((int)Value.TotalSeconds)) != Value.TotalSeconds;
 
            sb.AppendFormat("P{0}D", (int)Value.TotalDays);
            sb.AppendFormat("T{0}H", Value.Hours);
            sb.AppendFormat("{0}M", Value.Minutes);
            sb.AppendFormat("{0}S", 
                isFractionalSeconds ? 
                    "{0}.{1}".FormatWith(Value.Seconds, Value.Milliseconds) 
                : Value.Seconds.ToString());
 
            writer.WriteValue(sb.ToString());
        }
    }
 
    #endregion
}
 

I’ve taken this class and merged it into the System.Xml namespace – because clearly this will also work with the XmlSerializer as well as for the DataContractSerializer.

A few notes.

The nifty part of this class is in the use of the XmlSchemaProviderAttribute. This is the .Net framework’s preferred mechanism for mapping types to Xml schema. In theory, if you were writing a more complex custom type for which Schema simply cannot be auto-generated, you could manually inject the schema into the XmlSchemaSet passed into the GetTypeSchema method (the name of which is determined by the parameter you pass to the attribute constructor). You would then return the XmlQualifiedName of this schema type to satisfy the framework.

In this case all we have to do is to return the well-known qualified name of the xml duration type: ‘duration’ from the namespace ‘http://www.w3.org/2001/XmlSchema’. We should be able to rely on anyone working with XML to have mapped this namespace already, and we know that a schema exporter will be doing the same since most of the .Net fundamental types are mapped to the same namespace.

Under the hood I’ve written a simple regex parser based on the format for the Duration data type. It will recognise all valid strings, but it also lets one or two invalid ones through (notably ‘PT’). However, if you are also using schema validation then this will not trouble you.

As the comments state - years and months are a problem; since the .Net TimeSpan chickens out and doesn’t encode years/months (presumably because it makes it much easier to calculate them from the difference of two DateTimes, as well as to add one onto a DateTime). Of course neither have a fixed number of days; so I’ve gone for a reasonable average. You could be more clever and take the current month’s number of days plus a strict average of 365.25 days per year – it depends on how accurate you really need it to be.

If you’re writing a new web service, you can simply make sure that all your clients express durations starting with the number of days – e.g. ‘P400D’ which will be deserialized exactly into a .Net TimeSpan representing 400 days.

In situations where a duration is received from a client and might need to be sent back to them – the original input string is preserved (but it will be up to you to persist that server side).

So now you can change a DataContract class like this:

 public class Class1
{
    public TimeSpan Duration { get; set; }
}
 

And change it over to this:

 public class Class1
{
    public XmlDuration Duration { get; set; }
}
 

In this case these types aren’t annotated of course (whereas in my case all my exported types are).

If you’re publishing a data contract for a type that must also implement some internal interface that exposes a TimeSpan like this:

 public interface IClassInternal
{
    TimeSpan Duration { get; set; }
}
 

Then your best policy is to write the DataContract class as follows:

 public class Class1 : IClassInternal
{
    public XmlDuration Duration { get; set; }
 
    #region IClassInternal Members
 
    TimeSpan IClassInternal.Duration
    {
        get
        {
            //use implicit casting operator
            return Duration;
        }
        set
        {
            if(value!=null)
                Duration = value; //and here
        }
    }
 
    #endregion
}
 

Obviously, there is an argument here that the XmlDuration should, in fact, be a value type and not a class. I’ll leave that up to you to decide.

MVC Bug: The virtual path '[path]' maps to another application, which is not allowed

2010-09-16T22:15:00.001+01:00

I’ve being doing a lot of work on Asp.Net MVC (now v2) over the past few months and, firstly, I have to say that it totally rocks.

I will not go into any real detail about MVC here – this post is about the method System.Web.Mvc.HtmlHelper.AntiForgerytoken and that it highlights a bug in an internal class TokenPersister that you’re potentially going to get if you ever use spaces in your virtual directory names. Ironically, the class in question is marked with a comment that says it’s difficult to unit test because of the way it fakes Asp.Net requests – ironic that it should be such a class in which we find a bug! A better argument for unit-testing you’re unlikely to find.

Detail – Reproduce that bug

Using MVC it’s very easy to reproduce. I should start by declaring my system configuration:

Windows 2008 R2
IIS 7.5 installed and configured
Visual Studio 2008 with all service packs etc
.Net 3.5 sp1 plus security patches

VS2010 and .Net 4 are also on this machine – and I would welcome anybody following the same steps to confirm that i:

Create a new Asp.Net MVC2 Web Application (from the template) called ‘Asp Net Bug 2008’
- Don’t bother with the unit tests project
Open the project web properties, and set the project to use the Local IIS Web Server, the location should automatically be set to http://localhost/Asp Net Bug 2008/ (you might need to add the trailing slash here) – the spaces here are important so leave them in.
- Create the Virtual Directory
- Don’t forget to save the project file afterwards (like I just did as I created this walkthrough!)
Open the view Views/Account/Register.aspx
Just after the line
1. <% using (Html.BeginForm()) { %>
Add the following:
1. <%= Html.AntiForgeryToken() %>
Compile and run
Navigate to the [Log On] link at the top of the page
Hit the ‘Register’ link

You will see this exception helper:

And then on the Asp.Net error page, you’ll see this stack trace:

[ArgumentException: The virtual path '/Asp%20Net%20Bug%202008/Account/Register' maps to another application, which is not allowed.]
   System.Web.CachedPathData.GetVirtualPathData(VirtualPath virtualPath, Boolean permitPathsOutsideApp) +11193138
   System.Web.HttpContext.GetFilePathData() +61
   System.Web.Configuration.HttpCapabilitiesBase.GetBrowserCapabilities(HttpRequest request) +124
   System.Web.HttpRequest.get_Browser() +168
   System.Web.UI.Page.SetIntrinsics(HttpContext context, Boolean allowAsync) +207
   System.Web.UI.Page.ProcessRequest(HttpContext context) +232
   System.Web.Mvc.TokenPersister.CreateFormatterGenerator() +459
   System.Web.Mvc.FormatterGenerator..cctor() +10

[TypeInitializationException: The type initializer for 'FormatterGenerator' threw an exception.]
   System.Web.Mvc.AntiForgeryDataSerializer.get_Formatter() +38
   System.Web.Mvc.AntiForgeryDataSerializer.Serialize(AntiForgeryData token) +181
   System.Web.Mvc.HtmlHelper.GetAntiForgeryTokenAndSetCookie(String salt, String domain, String path) +405
   System.Web.Mvc.HtmlHelper.AntiForgeryToken(String salt, String domain, String path) +13
   System.Web.Mvc.HtmlHelper.AntiForgeryToken() +17
   ASP.views_account_register_aspx.__RenderregisterContent(HtmlTextWriter __w, Control parameterContainer) in c:\Users\andras.zoltan\documents\visual studio 2008\projects\aspnetbug2008\aspnetbug2008\Views\Account\Register.aspx:17
   System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children) +115
   ASP.views_shared_site_master.__Render__control1(HtmlTextWriter __w, Control parameterContainer) in c:\Users\andras.zoltan\documents\visual studio 2008\projects\aspnetbug2008\aspnetbug2008\Views\Shared\Site.Master:26
   System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children) +115
   System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children) +240
   System.Web.UI.Page.Render(HtmlTextWriter writer) +38
   System.Web.Mvc.ViewPage.Render(HtmlTextWriter writer) +89
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +4240

A closer look at this, you’ll see that the method GetBrowserCapabilities, called from the accessor of the HttpRequest.Browser is the culprit. Opening Reflector, we see that this method is attempting to load the site configuration (web.config presumably) – which is where GetVirtualPathData comes in. When I was confronted with this issue, it was obvious that the check which determines if the current request is in the current application root was failing. So the first thing I thought was to change our app root to be something more friendly.

The Fix

Take the spaces out of the virtual directory name; perhaps replacing with hyphens, or just zeroing them down completely. It’ll now work perfectly.

The Cause

Well, of course I haven’t analysed down to the last IL opcode here, but the first thing I thought was whether or not this was an Asp.Net bug, since the last few items in the call stack are from there. So you can try another test, if you simply add a standard WebForm to the project (making sure it’s hosted in a vdir again that has spaces in it), and put in the following:

<%= (Request.Browser!=null).ToString() %> 

Compile and run that, and you’ll correctly get ‘True’ output in the page. So, when the current HttpRequest is produced from the normal Asp.Net pipeline, everything works correctly.

So, let’s take another look at that stack trace – we have the call to

System.Web.Mvc.TokenPersister.CreateFormatterGenerator

Which the yields a call to

System.Web.UI.Page.ProcessRequest

But we’re already in that method further down, so why?

The reason can be found if you open CreateFormatterGenerator, either by getting the source code from CodePlex or by opening reflector. Let’s take a look at the method body (which I have ripped straight out of the published project from CodePlex):

public static Func<IStateFormatter> CreateFormatterGenerator() 
{ 
    // This code instantiates a page and tricks it into thinking 
    // that it's servicing a postback scenario with encrypted 
    // ViewState, which is required to make the StateFormatter 
    // properly decrypt data. Specifically, this code sets the 
    // internal Page.ContainsEncryptedViewState flag. 
    TextWriter writer = TextWriter.Null; 
    HttpResponse response = new HttpResponse(writer); 
    HttpRequest request = new HttpRequest("DummyFile.aspx", 
        HttpContext.Current.Request.Url.ToString(), "__EVENTTARGET=true&__VIEWSTATEENCRYPTED=true"); 
    HttpContext context = new HttpContext(request, response); 
  
    Page page = new Page() 
    { 
        EnableViewStateMac = true, 
        ViewStateEncryptionMode = ViewStateEncryptionMode.Always 
    }; 
    page.ProcessRequest(context); 
  
    return () => new TokenPersister(page).StateFormatter; 
} 

Well, the embedded comment there explains what’s going on – and it’s this fake request that’s causing the problem, clearly. Whatever it is that Asp.Net does to make sure we don’t get this error on a standard WebForm is not being done properly – although I’m not sure what that could be. My initial guess is that the %20 is being compared to the space in the path name, rather than being url-decoded first – and therefore a simple fix might be to change the call to HttpContext.Current.Request.Url.ToString() on the fourth code line above to pass in the url-decoded string. If I was feeling ambitious, I’d hack into the project and pursue a fix. However, my primary feeling is to report it to the Mvc team – which I have done here.

Serializing to attributes in WCF with DataContractSerializer

2010-09-16T22:14:00.001+01:00

It’s a common problem – you want to return an object from a WCF service as XML, but you either want, or need, to deliver some or all of the property values as XML Attributes instead of XML Elements; but you can’t because the DataContractSerializer doesn’t support attributes (you’re most likely to have seen this StackOverflow QA if you’ve done a web search). Most likely you’ve then migrated all your WCF service code to using the XmlSerializer (with all the XmlElement/XmlAttribute/XmlType attributes et al) – and you’ve cursed loudly.

Well, I’m here to rescue you, because it is possible – and the answer to the problem is actually inferred from the MSDN article entitled ‘Types supported by the Data Contract Serializer’.

The example I’m going to give is purely for illustration purposes only. I don’t have a lot of time, so work with me!

Create a new Asp.Net WCF service application, you can use Cassini as your web server (probably easier – otherwise you might have to enable Asp.Net compatibility mode).
Open the web.config and delete the <service> element that was created for the new service.
The interface and implementation model for this example is overkill. Move the [ServiceContract] and [OperationContract] declarations from the interface that was created for you new service to the class that was also created. Delete the interface.
Open the .svc markup file and add the following at the end: Factory="System.ServiceModel.Activation.WebServiceHostFactory" – this enables the zero-configuration WCF model for this service (we’re going to create a RESTful service).
Paste the following class declarations into your svc codebehind:

public interface IExampleData
{
	string Description { get; set; }
	string Name { get; set; }
	int ID { get; set; }
}
public class ExampleData : IExampleData
{
	public string Description { get; set; }
	public string Name { get; set; }
	public int ID { get; set; }
}
public class ExampleDataAttributed : ExampleData, IXmlSerializable
{
	#region IXmlSerializable Members
	public System.Xml.Schema.XmlSchema GetSchema()
	{
		return null;
	}
	public void ReadXml(System.Xml.XmlReader reader)
	{
		//implement if remote callers are going to pass your object in
	}
	public void WriteXml(System.Xml.XmlWriter writer)
	{
		writer.WriteAttributeString("id", ID.ToString());
		writer.WriteAttributeString("name", Name);
		//we'll keep the description as an element as it could be long.
		writer.WriteElementString("description", Description);
	}
	#endregion
}

Just to demonstrate the point, the class that will be part-serialized to attributes simply derives from one that will be serialized as normal.

Now add the following two methods to your service class:

[OperationContract]
[WebGet(UriTemplate = "/test1")]
public ExampleData Test1()
{
	return new ExampleData() { ID = 1, 
           Name = "Element-centric", 
           Description = 
"The contents of this item are entirely serialized to elements - as normal" };
}
[OperationContract]
[WebGet(UriTemplate = "/test2")]
public ExampleDataAttributed Test2()
{
	return new ExampleData_Attributed() { ID = 2, 
          Name = "Mixed", 
          Description = 
"Everything except this description will be serialized to attributes" };
}

Cover, and bake for 40 minutes (that is – Build it).

If you left your service as Service1.svc, then run it and open up IE and browse to http://localhost:[port of cassini]/test1

The result should look something like this:

<JSLabs.ExampleData 
 xmlns="http://schemas.datacontract.org/2004/07/ExampleNamespace" 
 xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
	<Description>
		The contents of this item are entirely serialized to elements - as normal
	</Description>
	<ID>
		1
	</ID>
	<Name>
		Element-centric
	</Name>
</JSLabs.ExampleData>

Now browse to http://localhost:[port of cassini]/test2

<JSLabs.ExampleDataAttributed id="2" name="Mixed" 
  xmlns="http://schemas.datacontract.org/2004/07/JobServe.Labs.Web">
	<description>Everything except this description will be 
	serialized to attributes</description>
</JSLabs.ExampleDataAttributed>

It’s made a little less impressive by that nasty ‘orrible “xmlns=” attribute that the WCF data contract serializer automatically puts on the type – but, as you can see, the ‘ID’ and ‘Name’ properties have indeed been pushed out as attributes!

We could have made both methods return IExampleData and then used the KnownType attribute on that interface in order to get it to support either (according to what the code of the methods returned).

To support deserializing an object from the attributes, all you have to do is to implement the IXmlSerializable.ReadXml method.

Finally, as the aforementioned MSDN article says about the supported types – you should also be able to use XmlElement/XmlNode types as a way of representing XML directly – the DataContractSerializer, like in this case, take the short route and simply gets the Xml.

This also shouldn’t affect JSON formatting if you’re dual-outputting objects for either XML or JSON clients.

Enabling Custom Username/Password Authentication WCF in IIS

2010-09-16T22:09:00.001+01:00

A couple of posts back I was bemoaning the inability to use custom username/password authentication for all endpoints (WsHttp, BasicHttp and WebHttp) for a WCF service hosted in IIS. The problem arises when you get to the WebHttp binding, because you can’t use message-level authentication (since it’s just a JSON packet), therefore you have to use Transport-level authentication. Then you get to the biggest issue - that using Transport-level means having to use Basic/Windows/Digest authentication and IIS always automatically intercepts those authentication headers and attempts to authenticate to the server’s default windows account store (typically the domain it’s on, or it’s own local user store).

In this case, even if you configure your Custom UserName Password validator, it’ll never fire.

My solution to do a full, roll-your-own solution using Message Inspectors etc. This still wouldn’t have solved the JSON problem fully, but it did mean that I could use my own Http headers to transmit my secure credentials.

I had a thought, though – what if IIS’ authentication could be extended, so that Basic authentication headers could be used on a per-endpoint basis (meaning that I was still free to use message-level for SOAP and Ws-Http if need-be)?

My initial google results were poor, and so I decided to use the MSDN Forums to get some ideas. Needless to say – the answer that Marco Zhou has been exactly the thing I was looking. All I did was to download the source for the excellent CustomBasicAuth project from CodePlex, and by following Dominick’s walkthrough was then able to extrude my own implementation that I could then plug into any WCF service.

The extra upshot of doing all this was that I can now use Role-based security on all my WCF methods, and the client just has to know how to send the credentials according to the type of endpoint they are using. With the JSON endpoints I’m using the trick I posted earlier on how to intercept web service calls from Asp.Net Ajax client code in order to inject extra headers, but with the SOAP endpoints (currently from Silverlight) I’m using message-level headers and a Message Inspector (TIP – if you follow the CustomBasicAuth walkthrough exactly, you’ll add an Authorization provider to your WCF service. If you then implement a message inspector to extract authentication details from a SOAP message, you can set the current Principal in that code – but here’s the beauty of it – it happens before WCF security kicks in, so the same Authorization provider will also work!).

So while it is a bit of a pain – once you’ve got the skeleton up and running, it’s actually pleasingly simple.

WCF URI Template Url-Encoded Character Issues

2010-09-16T22:07:00.001+01:00

I've been playing around with RESTFul services in WCF following by the MSDN topic, and have found the guide to be both extremely helpful, and the overall support provided to be excellent. If you've been writing WCF services for your cloud service offerings, or even just AJAX methods for web-page code, you should find the migration over to RESTful services relatively easy to manage.

It also increases the accessibility of your services a huge amount - very basic jQuery code can be used, for example, to bind to these services.

However, there are some issues surrounding the types of values for parameters that can be bound in either a path or query string segment - and it's all down to some url-encoded characters, and how they get interpreted (or rather not interpreted) before being serialised into method parameters.

The following example is taken from a Microsoft Connect bug that I have posted:

Setup a project to host a WCF .svc and create a method with two parameters:

[WebGet(UriTemplate = "{willfail}?willwork={willwork}")]

[OperationContract]

public string Foo(string willfail, string willwork)

{

Trace.WriteLine("willfail: " + willfail);

Trace.WriteLine("willwork: " + willwork);

string toreturn = 

	string.Format("willfail: {0}. willwork : {1}", 

			willfail, willwork);

return toreturn;

}

In my example, I set the service to be hosted in Asp.Net under IIS (optional, as it turns out) - and I'm using the zero-configuration WebServiceHostFactory factory in the .svc file.

Now, after building, start up a browser and browse to the url that the service is configured on - e.g http://localhost/[baseuriofservice]/[svcfile.svc]/c%23?willwork=c%23. '%23' is Url-Encoding-speak for '#'.

What you expect to see is "willfail: c#. willwork : c#".

What you actually see is "willfail: c. willwork: c#".

So, when url-encoded characters appear in path segments bound to parameters, they won't always be read correctly (incidentally '%20' is read correctly); but it appears that query-string ones will.

As I mentioned, I posted an Connect bug under the title 'WCF UriTemplate Strips Some UrlEncoded characters from bound path segment' - and it turns out that this behaviour doesn't just affect WCF, or even Asp.Net (since the MS investigator also tried self-hosting the service and it still fails).

Even better news, however, is that this bug has been marked as fixed - so hopefully in the next framework version, we can expect to be able to use path segments with the whole gamut of URL-encodings!

This is one in the eye for Ayende Rahien - who blogged very recently about Connect failing all over the place.

Adding custom HTTP headers with Sys.Net.WebRequestManager

2010-09-16T22:06:00.001+01:00

Following from my last post bemoaning the lack of transport-level client crediential authentication when running inside IIS, I've started implementing my own encrypted session-transfer between clients and the server.

I want to to use Http Headers as a primary way of broadcasting session information, and also because it's difficult for a hacker to know exactly what value to add to the http headers if it's been encrypted left, right and centre. I might extend this to using a cookie as well; but the advantage of relying on http headers is that it means clients other than web browsers will be able to access the services; so long as they can authenticate.

The primary focus of my proof of concepts at the moment are on the AJAX side, both Asp.Net Ajax and 'vanilla' Ajax (e.g. jQuery or direct XmlHttp) - since I know that in Silverlight, for example, I'll be using a different endpoint and can probably add the same authentication data to the message itself.

The scenario is this:

A service issues encrypted authentication 'tokens' which a client then sends to other protected services when it makes a request. These tokens are time limited and have to be refreshed by the client within a certain time period (possibility to make it request-limited as well)
A protected service is called by Asp.Net AJAX through use of the auto-generated proxy code that you the ScriptManager control gets - via the /js or /jsdebug endpoint that is added to the webscript-enabled service.

Now, the issue with this is that the proxy code that Asp.Net AJAX is generating is so 'friendly' that the underlying Web requests are abstracted away from your method calls, giving you no direct way of adding any custom headers you might want to add when you make the call.

Never fear - the Sys.Net.WebRequestManager class (part of Microsoft's Asp.Net AJAX client library) provides a hook which gives you the opportunity to examine each and every web request it sends, and even modify it's contents before it leaves the browser. In fact, setting up the handler is so simple, I'm almost embarrassed to have it sitting at the bottom of such a long preamble...

  1: //method which captures requests and adds headers etc

  2: function InterceptOutgoingRequest(sender, args)

  3: {

  4:   var currentRequest = args.get_webRequest();

  5:   currentRequest.get_headers()["MYCUSTOMHEADER"] = "MYCUSTOMVALUE";

  6: }

  7: 

  8: //adding the callback to the WebRequestManager

  9: Sys.Net.WebRequestManager.add_invokingRequest(InterceptOutgoingRequest);

The 'sender' parameter will be an instance of the Sys.Net.WebRequestManager, and the 'args' parameter will be an instance of Sys.Net.WebRequest.

Now, every single Asp.Net Ajax call made by the WebRequestManager will call your custom code.

WCF Transport-level Custom Username Authentication won't work in IIS

2010-09-16T22:04:00.001+01:00

Update – I have found a way (with a little help from the community!) around the issues discussed in this post – please see this post

I'm working on some WCF stuff at the moment whereby I want to expose business layer functionality to multiple clients (Javascript, Silverlight and WSHttp) and I want to have a security model that can be supported on all of them. My authentication is totally custom, since we have all of our users in a custom database.

Logically I want to be able to use the WCF security model since it is very rich, and automatic once you've done the configuration. However - it's a nightmare - because when you expose AJAX - friendly endpoints, you're doing it with WebHttpBinding, and it's security modes are, well, frankly crap; in fact, in almost every way WebHttpBinding should be thought of as a 'special case', which goes against everything that WCF seeks to achieve.

The problem is this - if you have a custom user name and password store, then you have to use either a membership provider (therefore relying on Asp.Net authorisation, and also in my case having to implement or stub out a whole host of features that our user store doesn't itself support), or you can write an entirely custom validator.

Once that's done your options - taken as a whole - are to use user name client credentials on either the transport or message security (using the 'clientCredentialType' attribute available on the <transport> and <message> security elements of a binding). You also need to wire up the custom username validator - which is done at the service behaviour level:

  1: <serviceBehaviors>

  2:   <behavior name="MyServiceBehavior">

  3:   <serviceCredentials>

  4:     <userNameAuthentication userNamePasswordValidationMode="Custom" 

  5:                             customUserNamePasswordValidatorType="YourType, AssemblyName"/>

  6:   </serviceCredentials>

  7:   </behavior>

  8: </serviceBehaviors>

In a wsHttp environment you can do anything - so I'll leave that one out.

In a SOAP environment, too, it's fine: You'd probably use Message-based security, so that you also have a way clear to enable transport-level security as well. Not to mention being also able to lock down the transport to further credentials if need be.

When, however, you hit the WebHttpBinding you also hit a massive brick wall. You can only enable Transport security or TransportCredentialOnly security. The first is just SSL, the second is no-SSL, but also gives space for the client credentials to be sent, using one of the Http authentication mechanisms, such as Basic, Digest etc (the same ones available on the IIS security tab).

This is where it gets really amusing - custom user name/password combos with Http Transport-based security do not work when hosted in IIS - because IIS steps in and authenticates against the windows user store (AD, local machine etc) automatically. For a double-kick in the teeth, it's not supported at all when the service is not self-hosted:

For confirmation of this - see this blog article, and this link on MSDN about the UserNamePasswordValidationMode property.

I'm now having to circumvent WCF security and am rolling my own based on an IDispatchMessageInspector implementation, and custom authentication headers.

The next post will be about how Asp.Net Ajax makes it super-easy to add extra headers to requests as they go out of the WebRequestManager class.

Using WPF to Render Bitmaps

2010-09-16T21:58:00.000+01:00

There are a few posts around the net about this, but a lot of them focus on taking XAML controls or pages, and turning parts of them into bitmaps.

This is fine, but I want to be able to use WPF entirely off-screen in order to create rich graphical content.

What I'm going to present here is a way of doing just that, in a console application just to press home the point.

This is the image that you're going to produce - it's a little bit, well, crap - but it does the job.

So, it's an Ellipse with a RadialGradientFill (off-centre, white-to-blue gradient stops), and a TextBlock in Arial Black/Bold font of around 40px size.

The edges around the ellipse are going to be output as transparent (open up this file in Photoshop or Paint.Net and notice the checkerboard patterns around the edge - useful for web images!

Step 1: Create the project

Open a new Visual Studio, and create a new Console Application Project. Make sure it targets .Net 3.5.

Add references to:

PresentationCore
PresentationFramework
WindowsBase

These are the core WPF libraries that are going to be required.

Warning - if you're going to be repeating this code in a Windows Forms application, you should be careful - because many of the classes in WPF clash with those in the standard windows forms framework.

Step 2: Basic setup

Now, a quick google or MSDN search will show that the main class you're going to need is called RenderTargetBitmap. This class has a method called Render, to which you pass a Visual. Anything that can be displayed in a WPF window is a visual (in fact the window itself is) which means, in theory, that anything you can do in WPF to screen can be done to a bitmap.

To produce the image file you're going to use one of the BitmapEncoder-implementing classes (there are classes for creating PNGs, Gifs, JPGs and more).

Open up Program.cs file and add the following usings:

using System.Windows.Media.Imaging;

using System.Windows.Media;

using System.Windows.Controls;

using System.Windows.Shapes;

using System.Windows;

using System.IO;

Modify the Main() method in Program.cs so it looks like this:

  1: [STAThread]

  2: static void Main(string[] args)

  3: {

  4: 	int height, width;

  5: 

  6: 	height = 300;

  7: 	width = 400;

  8: 

  9: 	//going to place all our stuff in here - same as on a standard WPF form.

 10: 	Grid mainContainer = new Grid();

 11: 	mainContainer.HorizontalAlignment = HorizontalAlignment.Stretch;

 12: 	mainContainer.VerticalAlignment = VerticalAlignment.Stretch;

 13: 

 14: 	Ellipse e = new Ellipse();

 15: 	e.Stroke = new SolidColorBrush(Color.FromArgb(255, 0, 0, 255));

 16: 	//setup an off-centre gradient fill for a 3D effect

 17: 	//Of course - we could instead use a Viewport3D with Visual3D elements :)

 18: 	RadialGradientBrush rFill = new RadialGradientBrush();

 19: 	GradientStopCollection gradientStops = new GradientStopCollection(new GradientStop[] { 

 20: 		new GradientStop(Color.FromArgb(255, 255, 255, 255), 0.0),

 21: 		new GradientStop(Color.FromArgb(255, 60, 90, 255), 1.0)

 22: 	});

 23: 	rFill.GradientStops = gradientStops;

 24: 	rFill.GradientOrigin = new Point(0.65, 0.25);

 25: 	rFill.Center = new Point(0.5,0.5);

 26: 	rFill.RadiusX = 0.5;

 27: 	rFill.RadiusY = 0.5;

 28: 	e.Fill = rFill;

 29: 

 30: 	e.HorizontalAlignment = HorizontalAlignment.Stretch;

 31: 	e.VerticalAlignment = VerticalAlignment.Stretch;

 32: 

 33: 	mainContainer.Children.Add(e);

 34: 

 35: 	TextBlock message = new TextBlock();

 36: 	message.Text = "WPF In a Bitmap!";

 37: 	message.HorizontalAlignment = HorizontalAlignment.Stretch;

 38: 	message.TextAlignment = TextAlignment.Center;

 39: 	message.VerticalAlignment = VerticalAlignment.Stretch;

 40: 	//use margin for placement

 41: 	//could use canvas instead - but don't like the static 'Canvas.SetLeft/SetTop'

 42: 	//methods.

 43: 	message.FontFamily = new FontFamily("Arial Black");

 44:    	message.FontSize = 40.0;

 45: 	message.FontWeight = FontWeights.Bold;

 46: 	message.Margin = new Thickness(0.0, 120, 0.0, 0.0);

 47: 	mainContainer.Children.Add(message);

 48: 

 49: 	PngBitmapEncoder encoder = new PngBitmapEncoder();

 50: 	RenderTargetBitmap render = new RenderTargetBitmap(

 51: 		width, 

 52: 		height, 

 53: 		96, 

 54: 		96, 

 55: 		PixelFormats.Pbgra32);

 56: 	render.Render(mainContainer);

 57: 	encoder.Frames.Add(BitmapFrame.Create(render));

 58: 	using (Stream s = File.Open("outputfile.png", FileMode.Create))

 59: 	{

 60: 		encoder.Save(s);

 61: 	}

 62: }

Notice a few things:

1) [STAThread] is applied to the Main method as WPF requires this. If you omit it, an exception is raised telling you as much.

2) Transparency is enabled in the output image by using a pixel format that includes an alpha channel. Line 55 specifies 32-bit rgba - same as most people's desktop.

3) Horizontal/Vertical Alignment being set to stretch in most places - has the same effect as it would on a XAML form.

Run the app and open up outputfile.png in the output folder.

Oops...

What's that? The image is empty?

What's happened is that, although the grid, ellipse and textblock are all present and, in theory, ready to be rendered, they haven't actually been told to lay themselves out according to their visual container. Of course, they don't actually have one, since they are just objects in memory, so what do we do?

We have to take the place of the layout engine that is usually wired up automatically by a real WPF form; which it achieves through the use of all the events that WPF elements expose (i.e. for when controls get added or updated etc).

On line 48 (or at least between 47 and 49) you need to add this line of code:

mainContainer.Arrange(new Rect(0, 0, width, height));

Thankfully, this is recursive.

Save and run again, and this time the image should be as above.

Animation

If you're going to try and create pre-rendered animations using WPF, there are a couple of gotchas - and one outright brick wall if animated gifs are your thing.

The brick wall is that GifBitmapEncoder is not capable of producing correct animatable gifs. While you can successively call the AddFrame method of it's Frames collection property, and create a single gif file containing all the individual frames, you can't (out of the box) add the necessary timing information and looping information that most browsers/image viewers require. You'll get a non-looping default-speed animation in IE7, for example, but in other browsers (apparently IE8 is one of them) it just won't animate at all.

It is possible to add this functionality into the Gif encoder - but it's not a job I want to take on (see this MSDN forum post and it's answer). In the meantime, it's still possible to use .Net and WPF to generate the intermediate gif frames on disk, and then either manually stitch the animation together with timing in a dedicated tool.

The gotchas for animating in this way are:

Once a BitmapEncoder has saved a file, it cannot be used to save another; so you'll need to create a new encoder for each frame.
You can re-use the RenderBitmapTarget between each frame, but...
In the same way that we had to manually call the 'Arrange' method of the Grid (or whichever root container we're going to be using), we also have to manually Invalidate any controls whose visual state changes between frames. You can achieve this, by sticking this line before the next call to the Render(Visual) method of the RenderBitmapTarget object:

mainContainer.InvalidateVisual();

Now, in theory it might be possible to use the story-boarding animation features of WPF, but since they rely on animation timers (which will not be present since there's no actual WPF app or window here) I'm not sure how effective it'll be. My guess is that you would manually visit the timeline, explicitly setting the current position from 0 -> 1, then invalidate anything that might be affected, and then do the Invalidate/Render code.

If that were possible, then it would certainly make animating rotations etc a lot easier.

In any case - the ability to render stills using WPFs frankly rather swish rasterisation layer is, in itself, a real boon if looking to create dynamic images on a web page, for example, or even for any other scenario. Automatic anti-aliasing and transparency, for example, being two of the biggest boons.

Passing ref parameters in a params object[] array

2010-09-16T21:51:00.001+01:00

Ever wanted to call a method that takes a ref something in a generic (but not using generics!) way, say through a params object[] parameter array?

I did, and was most frustrated that you can't.

The Scenario

Firstly - the scenario I encountered is pretty specialist - I'm dynamically generating types which override/implement base/interface methods with methods provided by XML files. The target methods of the implementations are static methods, and therefore cannot use the 'base' keyword when calling 'up' the virtual call chain. As a result, I have to have a common delegate through which the next method can be called. My solution to this is to have two standard delegates:

public delegate void ActionWithParameters(params object[] parms)

public delegate object FuncWithParameters(params object[] parms)

I like the params object[] route because it then means I can provide a public layer of methods, which are generics, which allow the developer to be more expressive with the types that they are passing to the next method, e.g:

//overload for two parameters

public void CallNext<T1, T2>(T1 p1, T2 p2)

{

	ActionWithParameters del = [Some Method to Get Delegate];

	//call it.

	del(p1, p2);

}

//the great thing being - for no parameters, it's the same code:

public void CallNext()

{

	

	ActionWithParameters del = [Some Method to Get Delegate];

	del();

}

However, this breaks down if one of the parameters is a 'ref'. We can overload the generic method with one or more 'ref' versions for each of the parameters coming in (lots of copying and pasting!), but then we get stuck when trying to invoke the delegate, which expects a params object[] array (incidentally, the same is true when using the InvokeMethod of MethodInfo).

Now, we can declare a delegate thus:

delegate void FooDelegate(ref string something);

Then we can bind new instance of the delegate to a method with the same signature, and call it:

public void MyFoo(ref string input)

{

	input = "Hello world!";

}

public void BindADelegateAndCallWith(ref string input)

{

	FooDelegate foo = new FooDelegate(MyFoo);

	foo(ref input);

}

So, obviously the inability to pass parameters by reference is not a limitation of the Delegate itself, it's a limitation of the means by which we can call it when not using it directly as a function (note - in the above example, the C# compiler redirects the 'foo(ref input)' call to 'foo.Invoke(null, ref input)' - calling a method which the compiler is auto-generating - open up Reflector and switch to IL mode on the 'FooDelegate' type).

Why?

Let's look at why we can't pass these reference parameters into a params object[] array - if you already know why, then just skip this bit!

All types in the .Net framework inherit from System.Object, right? Wrong. Typed references created using the C# keywords ref and out (i.e. 'System.String&' or 'System.Int32&') do not. In fact, they inherit from nothing, nor do they have any public/private members that can be discovered through reflection. Nevertheless, they are still discreet types in their own right. Indeed, to construct the 'Type' for a typed reference, would require something like the following code:

Type refstring = typeof(string).MakeByRefType()

Similarly, to get from the reference type back to the type to which it references you use the method:

//assuming that we still have the type from

//the previous example

Type nonref = refstring.GetElementType();

So, 'typed reference' types are not really part of the class hierarchy of a type and it's subtypes - but special types that exist in parallel to each class in that hierarchy - but which are only connected back to their non-reference bigger brothers. The same is also true for pointer types. So, for example, 'ref string' does not inherit from 'ref object' which is also the reason why, given a method like this:

public void PassObjectByRef(ref object obj)

{

}

You cannot call it with a line of code like this:

string mystring = "hello world!";

PassObjectByRef(ref mystring);

Again, the compiler will say something like "No overload which takes an argument 'ref string'", in addition to the other error "Cannot convert parameter 1 from 'ref string' to 'ref object'". The number of times I've been frustrated by that I can't count - you can of course sidestep this problem by having a temporary reference to the base class of the same object (i.e. declare an 'object' local variable, assign it to the local string, and then pass the object by reference instead).

So this explains why you can't pass a 'ref string' to a method whose signature is 'params object[]' - that notation indicates that the caller can pass zero or more parameters that have System.Object as a base-class and, as we've just seen, Type& does not.

The Solution

Now, the technique I'm about to show here is probably not the safest in the world - however so long as your code is tightly controlled it should be fine. It does require you to have an assembly somewhere which at least has the attribute:

[assembly: SecurityPermission(SecurityAction.RequestMinimum, SkipVerification=true)]

And if you're going to use this code, I'd suggest you place this attribute on the assembly in which you paste the code, because when it creates the DynamicMethods it associates them with the parent module of the type itself.

What we want to achieve is to be able to convert our typed reference parameter into a non-typed reference parameter on the caller side, push it through a params object[] function, convert that value back into a typed reference again, before passing it to the actual method - or even modifying the target reference indirectly through that value.

The code exploits the fact that a .Net typed reference is just a platform pointer under the hood and is, therefore, a value that can be stored in a System.IntPtr structure. It also exploits the fact that there are some things that we can do in IL that cannot be done in C# - like pushing typed reference's value on to the stack, and simply returning it without dereferencing it back to an object reference or value(the C# compiler automatically dereferences 'ref' parameters whenever they appear on the left or right hand side of an expression- either using OpCodes.LdInd_Ref or OpCodes.StInd_Ref, or similar, use Reflector or IL DASM to see what I mean).

Anyway, here's the magic generic class:

// Static class uses dynamic methods to do all the funky stuff, and

// because these are static they only get generated once for each type

// passed into the Type parameter T.

// Means that the first call will be a little slow, but the subsequent 

// calls should be lightning fast.

public static class RefParam<T>

{

	/*************************************************************/

	// The delegate types - these have to be declared using the

	// 'proper' delegate-keyword mechanism because only that

	// supports ref parameter types.  Would be nice to use

	// Func<T, IntPtr>, but we can't

	// Delegate which returns the actual object reference caused 

	// by passing the input parameter by reference.

	private delegate IntPtr MakeDelegate(ref T input);

	// Delegate which can be used to dereference an object 

	// reference and get the value currently stored in it.

	private delegate T GetValueDelegate(IntPtr objref);

	// Delegate which can be used to modify an object reference

	private delegate void SetValueDelegate(IntPtr objref, T newvalue);

	/*************************************************************/

	// this is the method that we actually call to turn a ref T

	// into an IntPtr.

	private static MakeDelegate _Make;

	// this method 

	private static GetValueDelegate _GetValue;

	private static SetValueDelegate _SetValue;

	static RefParam()

	{

		CompileDelegates();

	}

	private static void CompileDelegates()

	{

		//the dynamic method is targetted at the module that this

		//class is placed - that assembly needs to have the 

		//SkipVerification security permission.

		DynamicMethod dm = new DynamicMethod(

			string.Format("MakeRef_{0}", typeof(T).TypeHandle.Value), 

			typeof(IntPtr), 

			new Type[] { typeof(T).MakeByRefType() }, 

			typeof(RefParam<T>).Module);

		//method simply loads the reference value and returns it as

		//an IntPtr - if you were to disassemble any other method that

		//sucks in a ref parameter and returned it, you'd see a couple of

		//other opcodes - to do with dereferencing the reference - an 

		//operation that cannot be performed in straight C#

		ILGenerator gen = dm.GetILGenerator();

		gen.Emit(OpCodes.Ldarg_0);

		gen.Emit(OpCodes.Ret);

		_Make = (MakeDelegate)dm.CreateDelegate(typeof(MakeDelegate));

		dm = new DynamicMethod(

			string.Format("GetValue_{0}", typeof(T).TypeHandle.Value), 

			typeof(T), 

			new Type[] { typeof(IntPtr) }, 

			typeof(RefParam<T>).Module);

		gen = dm.GetILGenerator();

		gen.Emit(OpCodes.Ldarg_0);

		gen.Emit(OpCodes.Ldind_Ref);

		gen.Emit(OpCodes.Ret);

		_GetValue = (GetValueDelegate)dm.CreateDelegate(typeof(GetValueDelegate));

		dm = new DynamicMethod(

			string.Format("SetByRef_{0}", typeof(T).TypeHandle.Value), 

			typeof(void), 

			new Type[] { typeof(IntPtr), typeof(T) }, 

			typeof(RefParam<T>).Module);

		gen = dm.GetILGenerator();

		gen.Emit(OpCodes.Ldarg_0);

		gen.Emit(OpCodes.Ldarg_1);

		gen.Emit(OpCodes.Stind_Ref);

		gen.Emit(OpCodes.Ret);

		_SetValue = (SetValueDelegate)dm.CreateDelegate(typeof(SetValueDelegate));

	}

	public static void SetValue(IntPtr reference, T newvalue)

	{

		_SetValue(reference, newvalue);

	}

	public static T GetValue(IntPtr reference)

	{

		return _GetValue(reference);

	}

	public static IntPtr Make(ref T value)

	{

		return _Make(ref value);

	}

}

Some people might find bits of this code scary - from the point of view of safety (indeed, unless you associate the dynamic methods with a module within an assembly that has the aforementioned 'SkipVerification' permission enabled, these dynamic methods simply do not compile).

Now that we've got our static class, we can look at using it:

//copy this code into a standard unit test

[TestMethod]

public void TestRefPassing()

{

	//hello world!

	string input = "";

	TakesAParamsObject(RefParam<string>.Make(input));

	Assert.AreEqual("Hello world!", input);

}

//expects a single parameter to be passed of type IntPtr

//that can be converted back into a String&

public void TakesAParamsObject(params object[] parms)

{

	RefParam<string>.SetValue((IntPtr)parms[0], "Hello world!");

}

Run the test - it should pass.

So what if your params object[] method needs to take a parameter that is known to be a typed reference, and pass it to another method that expects a 'ref T'? This is a problem I've encountered in my project: even though I'm emitting the IL that wires this params object[] method to a 'proper' method with 'proper' parameters, I've not been able to produce IL that can safely turn an IntPtr back into a ref T without the runtime going monkey-poo (that's 'ape-shit') on me. I'm sure it can be done, but until I can figure it out, the simplest solution is to declare a local, copy the value of the object referenced by the IntPtr, pass the local by reference to the target method, and then write the local value back to the IntPtr reference afterwards. You can write C# to do the same - here's a slight modification of the above code:

[TestMethod]

public void TestRefPassing2()

{

	string input = "";

	TakesAParamsObjectAndForwardsItOn(RefParam<string>.Make(input));

	Assert.AreEqual("Hello universe!", input);

}

//same again, except this time the modification is being done in a method that

//actually expects a ref string, and the params object[] method simply cracks it

//out.

public void TakesAParamsObjectAndForwardsItOn(params object[] parms)

{

	//declare a string local, and use the 'GetValue' method on the 

	//RefParam<T> static type to retrieve the reference to copy into it.

	string topass = RefParam<string>.GetValue((IntPtr)parms[0]);

	//now pass the local string by reference

	TakesARefString(ref topass);

	//copy the reference back

	RefParam<string>.SetValue((IntPtr)parms[0], topass);

}

public void TakesARefString(ref input)

{

	input = "Hello universe!";

}

Now then, anybody out there who is also dynamically generating code to bind to methods like the 'TakesARefString' method through a similar params object[] method should be able to see how easy it would be to write the IL body for the middle method, given the target method's MethodInfo (being able to reflect the parameters and get their types).

Incidentally, I'm also using the same strategy for Value types passed by reference, and it all works fine.

What about garbage collection?

I'm pretty sure that, whilst this code might not be type-safe, it isn't 'unsafe' in the same way as using pointers. A typed reference in .Net incorporates information about an object within the managed heap (unlike a pointer which goes directly to the actual unmanaged heap) - so if the underlying storage is moved, the typed reference is still valid. If this wasn't the case, then any code which uses 'ref T' would also have to be unsafe, which it plainly does not. All we've done is to take the underlying data for the typed reference - which is basically just a smart pointer from what I can gather - and get it's actual value. There's no way that .Net writes back to these typed reference values if objects get moved about (far too much overhead, .Net would be a lot slower than it is), so we must consider the underlying value as sacrosanct.

Playing with fire might get you burned

Using this code is not without risks! I'm not going to bother with a disclaimer on this code; I'd be here forever. A few points of note, though:

We can now take that reference and pass it around outside of the call stack of the method that originally declared that reference - which the standard 'ref' mechanism prevents us from doing. Effectively, this can mean holding on to a reference to something that has since gone out of scope - usually a problem that you don't have to think about in .Net. Any attempt to do this (and then dereference the typed reference using the RefParam<T> class) is likely to end up with an ExecutionEngineException or something similar - and to get an idea of just how potentially dangerous that can be for the health of your application - consider the opening sentence for the remarks of the class from the documentation:

"Execution engine errors are fatal errors that should never occur"

Nuff said, really. When one of these occurs, you can't even 'catch' them they're that bad.

Also - DO NOT take an IntPtr reference made with, say, RefParam<int> and attempt to read or write it through RefParam<string> - that's really bad medicine. And don't even try to write to base types through a reference to the derived type - which as explained earlier the standard mechanism prevents you from doing. That said - I've not tried it... so it might work ;)

Beyond that, so long as you are diligent and ensure that use of this class is only used in place of typed references into method calls, you should not get any problems whatsoever.

Stopping the caller from passing in a reference type that you don't expect is another matter - however this can be achieved instead by wrapping the IntPtr in another type that encodes the original type of reference (something that the data referenced by the value of the typed reference encodes itself, but is unfortunately not something that you can get hold of), and then having your code check that before dereferencing it - throwing a much more friendly exception if it's the wrong one.

And finally...

If anybody's got any ideas how to inline the conversion from IntPtr back to 'ref T' for the purposes of passing to a 'proper' method, rather than having to use an intermediate variable, I'd love to hear from you - I have a feeling that I've missed something glaringly obvious.

Update (29th October 2008)

On this final point - I have solved the problem. This code is only going to be of interest if you are also code-generating a params object[] method which in turn calls a method with 'real' parameters - of which some could be ref params.

Here's a snippet of the IL I was using originally to load the original ref parameter (that was turned into an IntPtr using the RefParam<T> class) and store it in a local variable which then gets passed to the target method by reference in place of the reference that was passed (assuming that arg.0 is the params object[] array, and the IntPtr for the reference is the first item in it). I have not included the IL which synchronises the value back to the passed reference (using RefParam<T>::SetValue):

.locals init(
[0] paramelementtype param)
ldarg.0
ldc.i4.0
ldelem.ref
unbox native int
ldobj native int
call RefParam<paramelementtype>::GetValue(native int) //abbreviated the method name here for ease of reading
stloc.0 //storing the value in the local variable
ldloca.s param
//now call the target method

Now, when I originally tried to avoid using a local variable, I was experimenting with the refanytype and refanyval opcodes - thinking they would be of help. But no, they weren't, and so I just abandoned the idea in favour of that one above.

However, this morning I decided to take a look at it again - feeling sure that I could inline the conversion from IntPtr back to the Type&. My reasoning being that my GetValue method in RefParam<T> relies on this ability in order to work. Embarrassingly enough, the solution was simple - do away with all the extra code after the 'ldobj':

ldarg.0
ldc.i4.0
ldelem.ref
unbox native int
ldobj native int
//now call the target method

Why? Because after the ldobj instruction the 4/8 bytes of the managed pointer are now on the stack, and it is exactly that which the function call expects.

Generic overloads and ref parameter hell (the return of the Stack overflow)

2010-09-14T00:15:00.001+01:00

In a framework that I have written for my dev team, I have a class that has a method like so:

 public class ObjectBase
{
    public void Clone(ref ObjectBase target, ObjectBase parent);
} 
 

This method (virtual in my library, in fact) has two functions: When the target ref is null, then a new object should be created, but when it's not, then the contents of the object on which it's being called should be cloned into the passed object. The parent of the new object is required because in the framework our objects have keys that can come from multiple sources - including an expression on the parent.

Anyway, whilst this method is particularly useful, it's a little unfriendly to use from your code when you don't have a direct ObjectBase reference to pass, but instead a reference to an object derived from it:

 public class MyObject : ObjectBase
{
}
 
 
public static void main()
{
    MyObject obj = new MyObject();
    MyObject clone = null;
    obj.Clone(ref clone, null);
} 
 

That red line indicates where the compiler blows - when the parameter type is 'ref T', you can't pass a reference to 'TDerived : T', because although the two types are compatible, the method expects a 'ref T', not anything else, and at that level 'ref T' and 'ref TDerived' are two incompatible types.

Not a problem, we can of course get around this:

 public static void main()
{
    MyObject obj = new MyObject();
    MyObject clone = null;
    ObjectBase clonebase = null;
    obj.Clone(ref clonebase, null);
    clone = clonebase as MyObject;
} 
 

I don't need to say it, but I will anyway: URGH!

So, I thought, 'I know, I'll create a helpful generic method InObjectBase instead'. Here it is:

 public void Clone<T>(ref T target, ObjectBase parent)
    where T : class, ObjectBase
{
    ObjectBase basetarget = target;
    Clone(ref basetarget, parent);
    if (target == null)
        target = basetarget as T; //'class' generic constraint 
                                                            //allows us to use 'as'
}
 

Now I know the old adage about 'assume' making an ass out of you and me - but feeling very clever, I rolled out the code and started it up. I get a StackOverflowException almost immediately - and I'm concerned - because the code which uses the new generic version hasn't even been called!

So I go and investigate that method, and see that it is, in fact the second line - intellisense shows that it is now a recursive method. Even worse, every other call to 'Clone' (used perhaps about 1,000 times in many different projects at this point!) will now point to the generic version instead of the non-generic - meaning that I've just broken everybody's code.

Why?

The compiler matches methods initially by name, and then by signature (minus return type). It also factors in 'convertibility' from method call parameters to the types of the parameters in those methods that might be called, in case exact matches cannot be found. When the compiler performs a search for the target method of a method call, as well, it has to choose a sort order for the candidate methods when that method call is to an overloaded method. If you take a look at the work that the System.Reflection.Binder class has to do you'll get a good idea of this.

It would appear that generic method overloads are always placed at the 'top' of the sort order, meaning that they will always be used, even if there is a specific non-generic overload that matches the input parameters exactly.

Interestingly - this behaviour also seems to be the case even when the use of constraints on a generic type parameter effectively remove the generic method from being a possible candidate method for a call. My framework has an extra class on top of 'ObjectBase' which in fact any custom types would inherit from, so I modified the constraint on my generic to specify that type instead, and move the generic method declaration to that type (to avoid it appearing in misleading situations):

 public class ObjectCustomBase : ObjectBase
{
    public void Clone<T>(ref T target, ObjectBase parent) 
        where T : class, ObjectCustomBase
   {
     //can safely cast down to ObjectBase from ObjectCustomBase
     ObjectBase basetarget = target;
     Clone(ref basetarget, parent); //<--compiler error
     if(target == null)
        target = basetarget as T;  
   }
} 
 

This is where I start to wonder if there is a bug in the compiler, we get a compilation error on the line indicated in the comments with the following error:

The type 'ObjectBase' cannot be used as type parameter 'T' in the generic type or method 'ObjectCustomBase.Clone<T>(ref T, ObjectBase)'. There is no implicit reference conversion from 'ObjectBase' to 'ObjectCustomBase'.

Even though we now have an unambiguous call to the base 'Clone' method in ObjectBase - because its signature matches exactly the parameter types being passed in, and yet the compiler is totally ignoring it in favour of the generic, whose type constraint means that it should not be entertained at all. If anyone has any ideas how to sidestep this I'd be interested to here about them.

Annoyed

What's annoying is, whereas you can force a call to a generic method overload (say, for the purposes of intellisense or whatever) by explicitly specifying the angle brackets - but you cannot explicitly call a non-generic overload. Nor can I think of a suitable way that the language spec can be modified to include such a feature. Basically here we have a feature of the language that prevents us from doing something that would otherwise be trivial.

Solutions

There are two.

1) Silly, but allows us to maintain our intended design - reflect-invoke the non-generic overload of the 'Clone' method instead of directly calling it. We can cache the MethodInfo for it to avoid having to repeat those slow operations (see my previous posts on C# Meta-programming), however, it's a lot of work when we could instead just...

2) Change the generic method name!

Exception when using WCF Client to Web Service Reference behind a proxy

2010-09-14T00:03:00.001+01:00

Are you getting the following exception when calling a simple ASMX service via a WCF service reference from behind a firewall (in this case ISA Server):

"Server encountered an internal error. For more information, turn off customErrors in the server's .config file."

You'd expect there to be an inner exception, but there isn't. You can get to the actual exception by enabling all exceptions to cause a breakpoint in VS2005/2008 (Menu: Debug->Exceptions and then click the checkbox at the root of all .Net exceptions - be careful doing this in a web site, though, because a tonne of exceptions get thrown and swallowed by the Asp.Net engine!).

The actual error that is occuring is a WebException: [407] Proxy Authentication Required.

In my case, I've also been using Fiddler to debug the SOAP that's going to and from (if you haven't got it, then get it now!) my machine, and had also noticed that when it was running, the exception would go away. This is probably because of some behaviour on ISA Server where it will reuse existing credentials for a connection to a particular outgoing domain until a certain expiry time is reached - did throw me a curve however, until I figured that out!

Quick note - if you're looking at writing stuff to leverage TFS via it's web services and want to know how the Team Explorer and the Source Control Explorer do the things they do - it's a good idea to use Fiddler to examine the SOAP.

The problem here is an age old one where you have a proxy (in this case ISA Server) through which all your outgoing network traffic is performed. In my environment, the ISA Server requires credentials for a user on the domain that is allowed through the proxy. Without it, the connection will be refused, causing a RemotingException.

Now, in the past this could be solved using the System.Net.CredentialCache.DefaultNetworkCredentials property, and assigning it to a property of the SOAP client proxy that is generated - done. At first glance, there appears to be nothing as simple as that on the ClientBase<T>-deriving type that the WCF service reference generator in VS creates for you.

Looking closer, however, we see the member ClientBase<T>.ClientCredentials, which has a 'Windows' property, which in turn has a 'ClientCredential' property of type NetworkCredentials. Thinking that this would solve my problem, I wrote the following code:

 MyWebServiceClient client = new MyWebServiceClient();
client.ClientCredentials.Windows.ClientCredential = 
    System.Net.CredentialCache.DefaultNetworkCredentials; 
 

I should mention at this point, that the project I'm working on doesn't bind it's web service references using the WCF config files - because I have to bind to a dynamic endpoint. Anybody working with the configs, however, should still be able to use this post to correct this problem when using the config files.

Anyway, the exception still gets raised.

I then tried manually filling out the user name, password and domain, but that didn't work either [bangs head on wall].

Found this MSDN forum where the guy solved his problem using config. not a great use for me, given that I'm configuring declaratively but there is one nugget in there - where the guy sets the HttpTransportBindingElement.UseDefaultWebProxy property to false, and then specifies the address of the proxy in the HttpTransportBindingElement.ProxyAddress property.

This involved also using the CustomBinding class - which I don't want to use. I simply want to create a BasicHttpBinding object, set some properties, and have done with it.

So I tried switching off the default proxy behaviour, and then give WCF the address of the default proxy(!) - read it again if you can't see why I think this is ludicrous.

Here's my eventual code:

 [TestMethod]
public void TestMethod1()
{
    //create a binding
    BasicHttpBinding binding = new BasicHttpBinding
        (BasicHttpSecurityMode.TransportCredentialOnly);
    binding.Security.Transport.ClientCredentialType = 
        HttpClientCredentialType.Ntlm;
    binding.Security.Transport.ProxyCredentialType = 
        HttpProxyCredentialType.Ntlm;
    //don't read the system proxy
    binding.UseDefaultWebProxy = false;
    //in my case - the URL I placed here was the same 
    //as the one assigned by group policy! (WTF?)
    binding.ProxyAddress = 
        new Uri("http://proxy:portnum");
 
    //configure the endpoint address
    EndPointAddress addr = 
        new EndPointAddress("http://domain/service.asmx");
 
    //create an instance of the client type generated by the add 
    //service reference tool using the binding and the endpoint 
    //address that we've configured manually 
    MyWebServiceClient c = new MyWebServiceClient(binding, addr);
 
    //make sure that the network credentials are set on the client.
    ClientCredentials creds = 
        c.ClientCredentials;
    creds.Windows.ClientCredential = 
        CredentialCache.DefaultNetworkCredentials;
    //should bee true by default - but be safe.
    creds.Windows.AllowNtlm = true;
    //if the web service at the other end needs to use these credentials 
    //for impersonation or whatever, then this level should be set 
    //accordingly.
    creds.Windows.AllowedImpersonationLevel = 
        System.Security.Principal.TokenImpersonationLevel.Identification;
 
    //now call the method
    MyWebServiceClient.Foo();
}
 

And it works fine.

A quick note on this. I am a great fan of MSDN - typically my first port of call is the F1 help, go to the index, type the class into that, and then browse through. Typically you get some choice examples, with some extended documentation on the impact of different values of various settings etc. In some parts of the framework, you even find links out to high-level conceptual topics to see 'where you are' in the overall process of whatever that framework is trying to achieve.

Not with WCF - the F1 help for all the WCF classes is shoddy. It has to be the least well-documented significant framework since SMO (don't even get me started on that) - not joined up in any way.

Anyway, I hope this post helps somebody out!

More C# generic extension methods - non-interface interface methods

2010-09-13T23:49:00.001+01:00

(This blog post is copied from my old blog at www.lordzoltan.org, it was originally written 1st August 2008)

Somebody asked me the other day if there was a generic way of providing a property-based clone operation for classes. The operation didn't have to recurse into complex objects (just copy the references), but the value types had to be copied.

The first part of the solution was a little bit of reflection:

 public static void Clone(object source, object dest)
{
    //(omitted) check that the source and destination types 
    //are equal with Type.IsAssignableFrom
 
    PropertyInfo[] props = source.GetType().GetProperties(
        BindingFlags.Public | 
        BindingFlags.Instance | 
        BindingFlags.FlattenHierarchy);
    MethodInfo getmethod;
    MethodInfo setmethod;
    foreach (var p in props)
    {
        //make sure the get and set accessors are available.
        getmethod = p.GetGetMethod();
        setmethod = p.GetSetMethod();
        if (getmethod == null || setmethod == null)
            continue;
        //invoke the getmethod on the source, and use the result to pass 
        //to the set method on the destination
        setmethod.Invoke(dest, 
            new object[] { getmethod.Invoke(source, new object[] { }) });
    }
}
 

All great, so now we have a static method we can call - but, however nice it is, it's not that gawk-able.

I wanted to have this method appear in the member list of objects that we wanted to be able to clone - which meant either a base or an interface. Interfaces are less intrusive, because you can still apply them to classes with bases, of course, and you can have multiple interfaces - but having the method as an interface member didn't make any sense. However, if you've read my previous post: http://lordzoltan.blogspot.com/2010/09/pseudo-template-meta-programming-in-c_10.html, which uses some generic extensions alongside static generics to provide meta info, the solution becomes easy. However you have to do something that a lot of people would seriously moan about... an empty interface.

 public interface IGenericCloneable
{
}
 
public static class IGenericCloneableExtensions
{
    public static void Clone<T>(this T source, T dest)
        where T : IGenericCloneable
    {
        PropertyInfo[] props = 
            typeof(T).GetProperties(
            BindingFlags.Public | 
            BindingFlags.Instance |
            BindingFlags.FlattenHierarchy);
        MethodInfo getmethod;
        MethodInfo setmethod;
        foreach (var p in props)
        {
            getmethod = p.GetGetMethod();
            setmethod = p.GetSetMethod();
            if (getmethod == null || setmethod == null)
                continue;
            setmethod.Invoke(dest, 
                new object[] { getmethod.Invoke(source, new object[] { }) });
        }
    }
}
 

Notice that the reflection is performed on typeof(T) instead of source.GetType() - which should be faster at runtime.

There is a gotcha here, though, and it's to do with the T that is passed when this method called for base classes of a superclass, consider the following:

 public class Cloneable1 : IGenericCloneable
{
    public int ID { get; set; }
}
 
public class Cloneable2 : Cloneable1
{
    public string Name { get; set; }
}
 

Now some method declared in another static class, no attempt made to contrive a complicated scenario, instead we just have a rather pointless method, but it demonstrates the issue at hand:

 public static class Utility
{
    public static void GetClone(Cloneable1 source, 
        Cloneable1 dest)
    {
        source.Clone(dest);
    }
}
 

Now let's test the method for an instance of Cloneable1 and for Cloneable2:

 [TestMethod]
public void TestClone1()
{
    Cloneable1 c1 = new Cloneable1() { ID = 10 };
    Cloneable1 c2 = new Cloneable1();
    Utility.GetClone(c1, c2);
    Assert.AreEqual(c1.ID, c2.ID); //should be fine.
}
 
[TestMethod]
public void TestClone2()
{
    Cloneable2 c1 = new Cloneable2() { ID = 10, Name = "Fred" };
    Cloneable2 c2 = new Cloneable2();
    Utility.GetClone(c1, c2);
    Assert.AreEqual(c1.ID, c2.ID); //should be fine.
    Assert.AreEqual(c1.Name, c2.Name); //will fail
}
 

The second test will fail, because the GetClone method implicitly passes Cloneable1 as the <T> parameter for the Clone<T> extension method, since T is taken from the left-hand side of the '.' operator which selects the method.

One way to get around this problem is to write another extension method for IGenericCloneable in a new IGenericCloneableExtensions2 class (simplifies the reflection), this time which takes IGenericCloneable and is not generic, we can give it the same name because the compiler will intelligently select the method that matches the MOST derived version of the input parameters when it is called:

 public static class IGenericCloneableExtensions2
{
    public static void Clone(this IGenericCloneable source, IGenericCloneable dest)
    {
        typeof(IGenericCloneableExtensions).
            GetMethod("Clone", System.Reflection.BindingFlags.Static 
                | System.Reflection.BindingFlags.Public).MakeGenericMethod(
                    new Type[] { source.GetType() }).Invoke(
                        null, new object[] { source, dest });
    } 
}
 

Now we change the signature of our utility method to take instances of IGenericCloneable instead of the Cloneable1 class, and the compiler will redirect calls to the intermediate method instead of the generic.

This is a useful practise anyway, because it means you can make the IGenericCloneable(2) extensions available to code that only has instances of the interface (which, after all, is what most people will expect of an interface). However, we can bypass the need to regress the Utility.GetClone method to using IGenericCloneable instance, by tightening up our original generic version with a little bit of reflection and dynamic generic compilation magic:

 //modify the original version to check the 
//actual instance types of source and dest
public static void Clone<T>
    (this T source, T dest) 
    where T : IGenericCloneable
{
    Type[] instancetypes = new Type[] { 
        source.GetType(), dest.GetType() };
    
    if (instancetypes[0].Equals(typeof(T)) == false)
    {
        //check ultimate
        if (instancetypes[0].Equals(instancetypes[1]) 
                == false &&
             instancetypes[0].IsSubclassOf(instancetypes[1]) 
             == false)
            throw new ArgumentException("If the two instance types are not equal, " +
                             "the first instance type must be derived from the target type");
        
        ((MethodInfo)MethodInfo.GetCurrentMethod()).
                GetGenericMethodDefinition().
                     MakeGenericMethod(new Type[] { instancetypes[0] }).
                            Invoke(null, new object[] { source, dest });
        return;
    }
 
    //the rest is the same as before
    PropertyInfo[] props = 
        typeof(T).GetProperties(
        BindingFlags.Public | 
        BindingFlags.Instance |
        BindingFlags.FlattenHierarchy);
 
    MethodInfo getmethod;
    MethodInfo setmethod;
    foreach (var p in props)
    {
        getmethod = p.GetGetMethod();
        setmethod = p.GetSetMethod();
        if (getmethod == null || setmethod == null)
            continue;
        setmethod.Invoke(dest, 
            new object[] { getmethod.Invoke(source, new object[] { }) });
    }
}
 

Now, we can return our original GetClone method to take instances of Cloneable1, and the two tests should pass.

Don't be worried about the MakeGenericMethod call - the type engine will only actually compile the method once - after that, it will simply go and obtain a handle to the compiled method. [Note – with the advent of expression trees I now do things like this by dynamically compiling methods and then caching them – the result is far faster and much tidier :)]

There are other things we could do here, for example make another version of Clone which takes two type parameters:

 public static void Clone<T1, T2>(this T1 source, T2 dest)
    where T1 : T2
    where T2 : IGenericCloneable
{
    //same code as the original Generic one we wrote, 
    //using the property list of T2 instead of T1.
} 
 

This method automatically allows us to clone an instance of T1 to T2 because T1 is inherited from T2, and it'll still appear as an extension method to the instance of T1 because T2 is constrained to implement the IGenericCloneablee interface.

We could then modify the reflection code at the head of the modified generic Clone<T> method so that it picks up this generic method when the instance type of the destination object is a base class of the instance type of the source.

We could write a CloneNew<T> method, by specifying an additional constraint on <T>:

 public static T CloneNew<T>(this T source)
    where T : IGenericCloneable, new()
{
    T result = new T();
    source.Clone<T>(result);
    return result;
} 
 

And if you started stacking these empty interfaces on top of others, you can start building whole libraries of useful generic extension methods that provide a whole host of common functionality to otherwise method-less types - and still be able to inherit those types from concrete classes (because the interface list is unbounded).

Have fun!

Pseudo-Template Meta-Programming in C# - Part 2

2010-09-10T13:39:00.001+01:00

Following on from a taster that I posted before: http://lordzoltan.blogspot.com/2010/09/pseudo-template-meta-programming-in-c.html, here goes with the next piece.

Firstly, the design I'm showing here can be applied to any reflection scenario - I've deployed it in various guises, including for storing meta-data about methods in a class, properties, and even for compiling dynamic methods that provide late-bound functions. The primary reason why this design, I feel, is particularly useful is that you're making the .Net type engine do all the hard work of storing all this meta-data for you.

I'll go through this by looking at a really simple scenario:

You want to write a system whereby you can reflect properties of a type that a developer writes in order to provide some form of functionality. Let us say that you want to write a dictionary state-bag wrapper, where a property is wrapped around an internal dictionary of values. What you want is to have a code-generation friendly way for developers to add such properties to their class with the minimum of fuss. So, retrieving a property becomes as simple as:

 public string SomeProperty
{
    get { return base.GetValue("SomeProperty"); }
} 
 

You don't necessarily want to have the names in the state bag to be the same as the names of the properties, so you need to be able to allow the developer to specify, with the minimum of fuss, what the backer for the property is actually called.

The perfect solution to this is an attribute:

 [AttributeUsage(AttributeTargets.Property)]
public class DictionaryPropertyAttribute : Attribute
{
    public string Name { get; set; }
} 
 

Now we can use the attribute as follows (now making a class that we can use later):

 public class MyClass
{
    [DictionaryProperty(Name = "PROP_SomeProperty")]
    public string SomeProperty
    {
        get { return base.SetValue("SomeProperty"); }
    }
}
 

So far so good, but we need a base class to be called. What we're going to do here is to produce two levels of abstraction: an interface and a base class that we can inherit from. The base class will implement the interface to provide a simple implementation of a dictionary. Other implementations could wrap around a data row, ASP.Net ViewState or SessionState and so on.

 public interface IDictionaryProps
{
    void SetValue(string name, string value);
    string GetValue(string name);
}
 

For simplicity's sake we'll say that an implementation of GetValue must return null if a value with the supplied name doesn't exist. And SetValue should quietly overwrite existing values, while adding new ones.

Our base class, then, looks like this:

 public class DictionaryPropertyClass : 
    Dictionary<string, string>, IDictionaryProps
{
    public void SetValue(string name, string value)
    {
        this[name] = value;
    }
    public string GetValue(string name)
    {
        string toreturn = null;
        this.TryGetValue(name, out toreturn);
        return toreturn;
    }
} 
 

And that's it, now we can write the outer class, with a couple of minor adjustments, for the wrapped property that I outlined first, so that it inherits from DictionaryPropertyClass.

But wait, what about this mapping business? Well, that's where we enter the realm of static generics.

For anyone still getting to grips with generic types, they are basically classes with type parameters - unlike C++ templates, however, which are built at compile time in a template-style, the generic class itself (without any type parameters) is a type in its own right - except it cannot be used to create an instance.

TIP: For anyone already doing run-time wizardry with generics - try typeof(MyGeneric<>) - it returns a reference to the generic type rather than an instance of the generic type. If your generic has multiple parameters, then you simply include a comma for each additional parameter. typeof(MyGeneric<,,>), for example, returns a reference to the System.Type of a generic that takes three type parameters.

What we're going to do is to create a static generic class (i.e. we never actually create an instance of it) which is going to act as a container for the reflected property information for any type:

 public static class DictionaryClassMetaData<T>
    where T : class, IDictionaryProps
{
    //stores the mappings for properties to dictionary keys for T 
    private static Dictionary<string, string> _mappings;
 
    //obtains the mapping for a particular property by its name.  Caution, 
    //this is case-sensitive of course (use 
    //StringComparer.CurrentCultureIgnoreCase  in the constructor for 
    //_mappings to avoid this problem if case-sensitivity isn't required). 
    public static string GetKey(string propertyname)
    {
        if (_mappings == null)
            return null;
        string toreturn = null;
        _mappings.TryGetValue(propertyname, out toreturn);
        return toreturn;
    }
 
    //static constructor - a lot of debate about these - mainly because of
    //the dreaded TypeInitializationException when the constructor raises 
    //an uncaught exception, but if you are careful with your boot-strap code 
    //(and religious with try/catch blocks where necessary) you don't have 
    //anything to worry about. 
    static DictionaryClassMetaData()
    {
        BuildPropertyMap();
    }
 
    //the method that does all the work 
    private static void BuildPropertyMap()
    {
        //get the property list for T 
        //note the use of System.Linq here and a lambda expression that filters
        //the PropertyInfo objects returned from the GetProperties call so that 
        //we only get those properties which have our DictionaryPropertyAttribute 
        //note also the use of typeof(T) - allows us to reflect on the T that 
        //is fed into the generic type parameter of the static class. 
        //finally, note the use of the FlattenHierarchy binding flags - this 
        //means that our T can inherit from other types that also declare mapped 
        //properties and still be able to get the entire mapping for the whole type. 
        IEnumerable<PropertyInfo> props = typeof(T).GetProperties(
             BindingFlags.Instance | BindingFlags.Public | BindingFlags.FlattenHierarchy).Where(
                                    pinfo => pinfo.IsDefined(typeof(DictionaryPropertyAttribute), true));
        Dictionary<string, string> tocreate = new Dictionary<string, string>();
        DictionaryPropertyAttribute attr = null;
        //now to build our internal dictionary, if any properties were found 
        if (props == null)
            return;
        foreach (PropertyInfo pinfo in props)
        {
            attr = (DictionaryPropertyAttribute)pinfo.GetCustomAttributes(
                typeof(DictionaryPropertyAttribute), true)[0];
            //make sure the name is unique.  If so, add it. If not, then some form of 
            //error reporting should be performed. 
            //a good implementation should have a structure within this static type 
            //that can be used to obtain errors instead of 
            //throwing exceptions 
            if (tocreate.ContainsKey(attr.Name) == false)
                tocreate.Add(pinfo.Name, attr.Name);   
        }
        //if any properties found, set the dictionary, otherwise don't do anything 
        if (tocreate.Count != 0)
            _mappings = tocreate;
    }
}
 

Believe it or not, that's basically it. From outside of our class MyClass, we can now interrogate property mappings with the following line of code:

 string valuekey = DictionaryClassMetaData<MyClass>.GetKey("SomeProperty"); 
 

The first time a call like this is made, the .Net runtime will fire the static constructor for DictionaryClassMetaData<MyClass> (remember it's not a proper type until it's actually got a generic parameter in it!), after which it calls the GetKey method. After that initial call, however, it knows that it doesn't need to call it again, so it just fires the method. What's more, the .Net runtime ensures thead safety for that call, too, so you don't have to worry about locking anything in a multithreaded environment.

How does this help us with the initial plan? At the moment, we could reimplement our SomeProperty property in MyClass as follows (only the get accessor code is listed now):

 return GetValue(DictionaryClassMetaData<MyClass>.GetKey("SomeProperty")); 
 

Hmmm, smelly isn't it... What's more - the developer implementing their class needs to know about the meta class - which is no good.

This is where I admit that I've led us down a blind alley - the truth is that the original idea of using base.Foo(bar) as a methodology isn't going to cut it. Although, as you'll see in a minute, we could implement it that way, there is a better way that we can do it. Generic extension methods:

 public static class IDictionaryPropsExtensions 
{ 
   public static string GetMappedValue<T>(this T instance, string propertyname) 
         where T:class,IDictionaryProps
   { 
      string datakey = DictionaryClassMetaData<T>.GetKey(propertyname); 
      if(datakey == null) 
         return null; 
      return instance.GetValue(datakey); 
   } 
}
 

Now, we can rewrite our implementation of MyClass' SomeProperty as follows (this time, I'll include the full code):

 public class MyClass : DictionaryPropertyClass
{
    [DictionaryPropertyAttribute(Name = "PROP_SomeProperty")]
    public string SomeProperty
    {
        get { return this.GetMappedValue("SomeProperty"); }
    }
} 
 

Note how the implementation now uses 'this' directly. This is a strange feature of extension methods: when inside class scope, you will not see extension methods defined for that type unless you explicitly use 'this' - this is because without the 'this', the intellisense (and presumably the compiler) has no way of knowing that you actually mean 'this'. This is different from instance members - because they are within scope at the point where you are writing the code (when inside a class definition), therefore they are implicitly available.

We're also using the 'this' as the type parameter for our GetMappedValue extension method. If you were to breakpoint the extension method and do a get on the property, you will see that 'T' has been implicitly fed into it as MyClass. This is how we are then able to invoke the static meta class without any real effort whatsoever.

And finally...

Now that you have a static meta class that you can invoke in a single line of code, you can also do it with reflection. Consider this line of extremely generic code, which could be thought of as being similar to how the databound controls in Windows Forms, WPF and ASP.Net get hold of values when using the 'DataTextField' and 'DataValueField' properties:

 public string GetObjectPropertyValue(IDictionaryProps instance, string propertyname) 
{ 
}
 

So now we're faced with a slight problem - we can't talk directly to the meta data class, because we don't have a T to work with. If only we could dynamically invoke the meta class... Hopefully, if you didn't already know it, you have probably guessed that we can. This time we're using dynamic generics - and this is where my little tip earlier on of using typeof(SomeGeneric<>) comes in really handy:

 public string GetObjectPropertyValue(IDictionaryProps instance, string propertyname)
{
    
    Type t = instance.GetType();
    
    Type tmeta = typeof(DictionaryClassMetaData<>).MakeGenericType(new Type[] { t });
    
    return (string)tmeta.InvokeMember("GetKey",
            System.Reflection.BindingFlags.Public | System.Reflection.BindingFlags.Static,
            null, null, new object[] { propertyname });
} 
 

An interesting note about this code - even when you access a static class via reflection in this way, .Net will still maintain it's rule of firing the static constructor just once. Also, I'll wager that the speed with which it can obtain the type instance after the initial call is pretty good.

You can (and I have) cache the method for future use in a Delegate stored within the instance (think 'Delegate.MakeDelegate(typeof(Func<string, string>), reflectedmethod);') - this speeds up future access a great deal, because the JIT can optimize away a lot of the method-jumping.

There are some truly crazy things I've done with this model and I might blog about them in the future when I get a chance - in the meantime I hope this provides satisfaction for your coding needs!

Pseudo-Template Meta-Programming in C#

2010-09-10T13:04:00.001+01:00

Not full details here, but I'm writing this post as part-memo to myself, but also in the hope that one or two of you might make a mental bookmark to keep checking back.

In the course of my most recent project, which involve classes which wrap a common data storage mechanism (a bit like DataRows) I encountered a problem which was best solved with a solution called (coining a phrase) 'self-aware' properties. Assuming all the storage access code is provided in a common base, and using attributes to define the string name of the 'column' (to continue the DataRow analogy) to which a property is mapped:

class X
{
[MappedProperty("actualvaluename")]
public string SomeValue
{
     get
     {
        return base.GetPropertyValue("SomeValue");
     }
}
}

So far so rudimentary. However, I wanted my framework to be able to support inheritance, so that a class inheriting from another one not only could inherit it's properties, but also so it could actually override the mappings for a property defined by a base, just by using the same attribute.

Also, what I didn't want to have to do was:

a) Store mapping information in every instance of such a class (to weighty)
b) Write some behemoth registry which keeps track of all the mappings (initialisation problems, potential threading issues, not to mention a nightmare to extend and maintain)

The solution that I have produced takes a few ideas from the ideas laid out by C++ Template Meta-Programming (read this book if you want to know more: http://www.amazon.co.uk/Template-Metaprogramming-Concepts-Techniques-Beyond/dp/0321227255), but of course it all happens at run time - because Templates in C++ are not the same as C# Generics. It takes advantage of an immensely powerful feature of Extension Methods - generic extension methods (think public static void Foo<T>(this T instance, ...) ), as well dynamic Generic class instantiation using reflection to achieve the generic recursion required to build the data for each level in the class hierarchy.

All it took was a common base (with practically no methods), a static generic class plus some extension methods - and the C# type engine does the rest for you.

In a future post I will lay out the design, but for now I've run out of time (it's Fathers' day - which as many fathers will know doesn't necessarily mean that Father gets to choose what he does!)