Home - Nadeem's SharePoint Blog

3/21/2008

Problem Solved: Generating Dynamic CAML Queries

I've turned the process of writing CAML queries into a piece of cake.

The solution is a very simple helper library that has zero parsing/conversion code that could go wrong, and it's all based on a simple idea (more on that later).

The solution is especially useful if you need to generate a CAML query dynamically. This is where conventional tools like the U2U CAML Query Builder fail.

By dynamic CAML query, I mean that the nature of the query (the combination of fields that will be used) is not known at compile time. This is usually the case when it is up to the users to select the filter/query criteria that they want to use.

The idea of a dynamic CAML query has huge applications in reporting for almost any type of SharePoint application (so long as there is data to be found in SharePoint lists or document libraries). By eliminating the difficulties in creating CAML XML queries dynamically, it becomes relatively easily to create a simple "advanced search"-like form that users can use to view a roll-up of all items that match their given criteria. This roll-up report or view would be real-time and accurate in that it would have zero dependencies on your search crawl schedules or search configuration.

An example is necessary to clarify. Suppose I wanted to find all default.aspx pages and Word documents that were modified today and created within this year. Sounds simple enough, right? The CAML query XML might look like this:

<And>
<And>
    <Eq>
      <FieldRef Name="Modified" />
      <Today />
    </Eq>
    <Gt>
      <FieldRef Name="Created" />
      <Value Type="DateTime">2008-01-01T00:00:00Z</Value>
    </Gt>
</And>
<Or>
    <Eq>
      <FieldRef Name="Name" />
      <Value Type="String">default.aspx</Value>
    </Eq>
    <Contains>
      <FieldRef Name="Name" />
      <Value Type="String">.doc</Value>
    </Contains>
</Or>
</And>

(I realize the example isn't exactly something very practical but please bear with it. I'm also quite sure you can check the file extension using a more specific FieldRef but I wanted to show "Contains").

With my solution, it's a matter of writing the following C#:

Caml camlQuery =
   Caml.FieldRef("Modified") == Caml.Today
   & Caml.FieldRef("Created") > Caml.Value(new DateTime(2008, 1, 1))
   & (Caml.FieldRef("Name") == Caml.Value("default.aspx") | Caml.FieldRef("Name").Contains(".doc"));

To get the current XML string for the camlQuery object (e.g., for use with SPQuery or SPSiteDataQuery), simply use camlQuery.OuterXml or camlQuery.GetFormattedString().

For one thing, the above C# code is much easier to read and write than the corresponding CAML XML. In addition, you can add clauses dynamically as necessary (e.g., depending on whether a textbox has a value or not). As a bonus, you get Intellisense and there is little possibility of a typo.

If you've ever tried to build a CAML query dynamically in a manual way, you probably quickly realized that you were getting nowhere fast and it may have seemed like you had a very complex problem to tackle.

This problem is the direct result of the way CAML uses the fast-to-process but unintuitive prefix notation instead of the infix notation that is natural to human beings. In other words, we humans find it awkward to state a logical (or mathematical) operator before its two operands. We would rather say "Cat OR Dog" and "3 + 4" instead of OR(Cat, Dog) and Add(3,4). It starts to get more and more absurd for us when the result of an operation is the argument of another -- consider the ease with which we can write and understand

(8 + 2)/5 x 4

versus

Multiply(Divide(Add(8, 2), 5), 4).

There is a systematic way to take your query from an intuitive infix representation to CAML's prefix form and you knowingly or unknowingly do this conversion when you need to write a CAML query, but what if you're not there to write the query XML when it's needed? What if your code has to generate the query XML based on specified web part properties or based on the user's submission of an advanced search form?

Side note: I am not saying that CAML should have been written to use infix like SQL. Computers like to know the operator (function) first and then the operands (arguments) second, and when you write infix you just create a lot more work for the computer. How would you accomodate infix with XML? You really can't. And if you didn't use XML, the computer probably have to be a lot of manual string parsing. Finally, in the end the computer probably needs everything converted to the function-followed-by-arguments form and so it needs additional (unnecessary) work to get there.

So does my solution convert infix strings to postfix CAML XML? Actually, no. As I said, there is zero parsing code involved and the solution is much simpler than that and also less error prone. All I do is simply create a class called Caml that inherits from XmlElement and I override all of the binary operators (==, <, >, <=, &, |, etc) so that they take Caml objects as arguments and return a 3rd new Caml object (which is, of course, just more Xml). I let the C# compiler do its magic that allows you to write a binary operator like '<' between its two arguments. Then you can combine all of the clauses and logical operators in your query just as intuitively as you write an "if" statement.

See Nadeem.SharePoint.zip (in the Documents library of this blog site) for the code. There are two flavors available: the initial version that I wrote very quickly as a proof-of-concept using the Nadeem.SharePoint namespace (in which all code is in one simple .Cs file), and a version using the Infusion.SharePoint namespace that was very nicely refactored, commented and cleaned up by Szymon Rozga, a colleague of mine at Infusion Development.

Side note: I realize that LINQ 2 SharePoint (formerly LINQ 2 CAML) addresses difficulties associated with writing complex CAML queries, but not everyone is able to deploy .NET 3.5. Moreover, I am not sure how easy it is to create dynamic/ad-hoc LINQ queries (I admittedly do not have any LINQ experience).

Posted at 4:31 PM by Nadeem Mitha | Category: Windows SharePoint Services 3.0 | Permalink | Email this Post | Comments (1)

4/3/2007

Toronto Code Camp Session Follow-up: Rapid Web Application Development Using Windows SharePoint Services 3.0

For those of you who attended, I hope you enjoyed my talk at the Toronto Code Camp last Saturday. Click here for the slide deck.

I will be delivering the talk again at the next Toronto SharePoint User Group meeting, which should be occurring sometime this month (visit http://tspug.sharepointservers.com/ and subscribe to alerts to stay up-to-date with the user group). Hopefully I will have working demonstrations this time around.

Outline of talk:

Discover the advantages of leveraging SharePoint as a platform for web-based applications

Learn a tried-and-tested approach to extending SharePoint lists in order to facilitate complex entity relationships

Find out how to customize various out-of-the-box behaviour to get what you need out of SharePoint for your web applications

Posted at 5:57 PM by Nadeem Mitha | Permalink | Email this Post | Comments (1)

1/29/2007

Building a better SharePoint blog

I've been spending some spare time improving the OOTB blog site in SharePoint 2007 (WSS 3.0).

Here's what I got working so far on our development server:

Trackback support. Send or receive trackback "pings" with your SharePoint blog.
Assign multiple categories to a post (this is too easy to do.. why wasn't it this way OOTB??).
Links at the bottom of each blog post to add the post to del.icio.us, favorites.live.com, digg.com and reddit.com.

At the risk of this blowing up in my face, I've decided to be bold and give you a direct link to a test site on our development server: Enhanced SharePoint Blog. I make changes directly on the server and perform iisreset all the time, so it may be off and on or it may be broken while I'm working on something new.

I might divulge more nitty gritty details later, but for now I'll just quickly go over what I did to get these "features" working, except I'll let you figure out how to implement "many to one" category assignment.

For trackback support:

I have an aspx page in the _layouts directory that acts as a trackback ping receiver (see the trackback spec). By using a querystring parameter for the Post ID, the single aspx file meets the spec requirements for a unique trackback ping URL for each and every post. If you HTTP POST the aspx "page" with the proper variables, it will register the received trackback ping in a custom list and deliver the appropriate XML success response back to the pinger.
Each post shows a count of the trackbacks it received along with a link to view the details. I used the OOTB "count related" capability of the lookup column to create a "# Trackbacks Received" field, and then I have another computed column that uses a simple CAML DisplayPattern to show the first column within a hyperlink to a filtered view of the received trackbacks list. (I see how that might sound complicated, but really it's not...).
A special computed column renders (invisible) XML RDF meta tags in the post's HTML to expose the trackback ping URL. There is also a computed column linking to a simple aspx page that lets you grab the trackback ping URL when you're blogging platform doesn't support the ping URL auto-discovery (nothing I've found seems to support this part of the spec, except for Movable Type, the spec's creators. Even I didn't bother implementing auto-discovery yet since no blogs expose the required metadata anyways).
Another list stores "outbound" links (trackback or not). Another computed field renders a hyperlink to add an outbound link for a particular post. The hyperlink includes a querystring parameter rigged to some JavaScript on the NewForm.aspx page in order to preselect the post ID based on where you clicked the hyperlink. I have yet to create a simple event receiver to parse through the post body text and extract all external hyperlinks. So for now, performing a trackback ping is a manual operation.
When you add an outbound link, an event receiver will perform a trackback ping to the other person's blog if you specified a trackback ping address. Again, I didn't implement the auto-discovery part of the spec so you have to specify a trackback web address in addition to the permalink (but show me a popular blog implementation that actually exposes the required metadata and I'll do it). Note (May 8, 2007): I have since discovered that many blogging applications will look for a <link> tag that specifies what the URL is for sending trackback pings.

For the del.icio.us and other links:

These are just computed fields with a custom DisplayPattern to output the HTML for the necessary hyperlinks. Note: there's a big difference between "Calculated" and "Computed" fields. Since each of these fields pretty much has the exact same CAML markup, perhaps one day I'll create a custom field type so that I don't have so much ugly copy-and-paste in my site definition. Better yet, this would let you add your own "URL action"-type fields (i.e., read-only hyperlinks with certain variables in them like item URL and Title).

I made all of these changes directly in site definition files. The SharePoint feature framework doesn't make it easy to do the things I needed. With features, you cannot add a column to an existing list definition/schema. You also cannot modify the existing views specified in the list definition to show these fields. I suppose your feature could add site columns and then a .NET feature receiver assembly could tack the columns onto the necessary list and views, but this just seemed like too much effort to be worth it. Also, not only would development be slowed down but the site itself would be slower due to schema differences being loaded from the database and merged with the site definition.

Perhaps another approach would be to create a content type feature that duplicates the OOTB fields in the Posts list and then adds in the custom columns. You would then have another feature bind the first feature to the Posts list as the default content type. But after all that effort, I'm not even 100% sure this would work. And I'd still have to do something about getting the new fields into the views.

Finally, here's a quick list of further enhancements I'm planning for SharePoint blogs:

Pingback support: another "Linkback" standard in addition to Trackback
Using the Content Query Web Part to aggregate blogs in subsites. I started this at http://home.infusionblogs.com, but I need to make the links work for anonymous users and use a custom XSLT to display more information like the name of the blogger.
Adding shortcut links as part of the new blog site definition. These shortcuts would allow you to easily subscribe to e-mail alert notifications for posts, comments and trackback (as well as any other handy shortcuts I can think of).
Providing the ability to rate each blog entry. Right now I'm thinking of tying the Posts list to a custom survey.
Finding out why the Google "Subscribe" bookmarklet doesn't work, and then fixing the issue.

I'm also very open to suggestions for how to make SharePoint blogs that much better. By the way, who thinks I should distribute the site definition when I'm done?

Posted at 6:08 PM by Nadeem Mitha | Category: Windows SharePoint Services 3.0 | Permalink | Email this Post | Comments (0)

The mind of a coder

My last post motivated me to reiterate a point I tried to make. In that post, I said:

"You have to realize that code is not something magical that happens to do what you want as long as it's written in a certain way. When you code, you are providing instructions to something that can't read your mind and doesn't know what you want to do. So in your code you have to be explicit about the things you intend to do and what exactly it is that you're doing these things on. So if the code doesn't, in some way, incorporate an instruction or piece of information that anything or anyone would need in order to do the right task on the right thing, it couldn't possibly be correct."

In my experience, I find that some people "get it" and some people just don't. The latter, I'm afraid, don't make good coders. It's certainly not my intent to offend anyone, I'm only trying to say that some people don't seem to possess a certain manner of thinking that I think is crucial to a good coder. And that's fine - nobody excels at everything. If you're not a good coder, you're probably a good something (and whatever you're good at, chances are it can help you with your social life moreso than coding ever could).

I was once helping out a friend with a physics problem. It was something like figuring out how far an object went after being thrown straight off the top of a building X meters tall at a speed of Y meters per second. The person was frustrated and somewhat astonished that the right answer wasn't "just coming out" of a formula even though they never specified what the height of the building was anywhere.

I find that a common approach to problem solving (being physics or coding) is to mash and mangle anything that might make sense until you get something that seems to be correct. I came from a program where a lot of people were forced to take courses in programming. They hated it, claimed they "didn't get it" and admitted they were very bad at it. Here is what I often noticed about their approaches and beliefs regarding coding (perhaps being cogniscent of these kinds of things might make you a better coder):

They say things like "It just needs to be this way. That's how it works, I don't know why. Just do it."
They say "It's not doing what it's supposed to" (as if "it" is supposed to know).
They almost always blame their hard time with coding on not knowing syntax. Obviously, everyone knows the difference between syntax and logic but some people just don't seem to be conscious of this difference all the time.
They get overwhelmed by bigger tasks instead of dividing and conquering.
They try to put together different lines of codes and different constructs that they learned assuming that they'll get something to work. "Let me try that while loop thing and just put this part in there and see if that makes it work...". (I swear to you I've encountered this. No further comment).

So how do you think like a coder? I don't know... I think you just do, or you don't.

Posted at 4:50 PM by Nadeem Mitha | Category: General | Permalink | Email this Post | Comments (2)

TS Exam 70-541: WSS 3.0 - Application Development

I took this exam on Friday and I want to share the following with anyone who is interested in taking it:

For a limited time (while it's still in beta), you can take the exam for free using promotion code BTA541. You save $125 (and you don't have to deal with your company's beloved expense-tracking and reimbursement system). Register at www.prometric.com.
Don't bother looking for practice exams or Microsoft e-Learning courses that will help you prepare for this. There isn't anything as far as I can see.
Ishai Sagi has a blog post with some tips for this exam and the MOSS equivalent (70-542). I also need to thank Ishai for posting the promo code.
Don't rush. There's more than enough time to think about each question thoroughly.
There seemed to be a lot of questions on Records Management, which I wasn't expecting because that's more of a MOSS thing. But I guess it integrates with WSS-only spaces, so they saw it as fair game. On the other hand, I didn't notice there being a lot of questions on custom field types like Ishai mentioned.
It looked to me as though the exam is in beta for some good reasons. I am very sure I saw questions where the only sensible answer had a typo or two in the code. And I suppose I could be wrong but I swear I looked at some answer choices line by line and couldn't notice any difference except a variable name. For example, in at least one case it looked as though the only difference in the choices was a change from the word "site" to "web" but only in the variable name (they probably meant to change the object type too but missed that). So if the exact same code appears in two different choices for the answer, what if I wanted to pick that code.. which one do I choose?

I found that when writing this exam, the following (which is true in general) is especially true in this case: You can often see to it that you probably get the right answer even if you get to a question where you've never seen the code before. Maybe a question is asking about a certain part of the SharePoint object model that you have never encountered in your work, or whatever may be the case. You really cannot (or at least you should not) be completely prepared for any possible code question. It would be a huge, wasted effort if you memorized all aspects of the SharePoint OM and CAML syntax.

So what do you do? Answer what you can where you know your stuff, and then note the following to help you with the rest:

Sometimes the code in a multiple choice answer selection simply doesn't make sense. You have to realize that code is not something magical that happens to do what you want as long as it's written in a certain way. When you code, you are providing instructions to something that can't read your mind and doesn't know what you want to do. So in your code you have to be explicit about the things you intend to do and what exactly it is that you're doing these things on. So if the code doesn't, in some way, incorporate an instruction or piece of information that anything or anyone would need in order to do the right task on the right thing, it couldn't possibly be correct. I hope this is clear.
It may not hurt to be a little subjective about the code that is presented to you as possible answers to a question. Ask yourself things like:

"If I was Microsoft (or VTI, I suppose) and I created this object model, would I really expose this method on this object with these parameters in this way?".
"Is this good code?" (of course, it doesn't have to be in order to be the right answer... But read the following.)

If a possible answer to a question is built on an object model or markup language syntax that really deviates from standard best practices (like using enums, following the OOP paradigm, etc) then it's possibly more likely to be a wrong answer. I find that "embarassing" examples of a programming model or syntax are usually conspicuously absent from SDK documentation, learning material and other publications that a company puts out. It's usually us bloggers that find and expose these sorts of things.
You don't need to get every answer right. So don't worry about it and move onto the next one.

Now it's time for the MOSS exam. This one's going to be tough...

Posted at 3:45 PM by Nadeem Mitha | Category: Windows SharePoint Services 3.0 | Permalink | Email this Post | Comments (0)

1/26/2007

Coming soon to this blog...

I don't yet have a fancy title at Infusion such as Practice Manager of Emerging Technologies, but I've been working with SharePoint products and technologies for some time now and there are many tips, tricks and customizations that I want to share with the community.

Here are some of the things I've been busy with that I hope to write about at some point or another:

Writing beta TS exams for WSS v3.0 and MOSS 2007 Application Development. I completed 70-541 earlier today and I have a few things to say about it (like what is with the questions where answers A and C would have the exact same code, only with different variable names??).
Preparing a presentation on Rapid Web Application Development in WSS v3.0. Click here for the outline. I plan to deliver the presentation at an East of Toronto .NET User Group meeting to be held on March 15. If nothing else, at least take a look at the outline for a list of things you get for free (without coding or writing config files) when using WSS as a platform for your web apps.
Working with Kurt to make SharePoint's plain vanilla blog support a lot more interesting and useful. For example, on our dev environment I've added trackback support that is nearly compliant with the full trackback spec. This means you will be able to write about an Infusion SharePoint-based blog post from your blog (on a supported blogging platform such as Live Spaces) and the Infusion blog post will automatically acknowledge your "trackback ping" and link back to your blog (and, of course, the Infusion blogger can subscribe to alerts to get notified when this happens). Other things I want to add are simple actions such as "kick it" and "bookmark it" (easy) as well as ratings for each blog post (a bit more tricky). (By the way, all of this customization involves fun stuff like computed fields specified in CAML list definitions, the SharePoint object model and little known features such as the "count related" capability of the lookup column).

In addition to describing technical "adventures" and findings like the above, I might also write about things like the business implications of using SharePoint and my experiences with how and why companies are embracing (or not embracing) SharePoint. (Is it just me, or do you also find that organizations sometimes absolutely love SharePoint even though they have no idea how to make the most of it?)

Posted at 4:26 PM by Nadeem Mitha | Category: General | Permalink | Email this Post | Comments (0)

RSS Feed

Nadeem's SharePoint Blog

Nadeem's SharePoint Blog

3/21/2008

4/3/2007

1/29/2007

1/26/2007

Shortcuts

‭(Hidden)‬ Admin Links

Latest Entries

Links

Documents