jQuery makes JavaScript suck less.  KRL uses jQuery selectors extensively in actions to position, modify and insert elements into web pages.  It is a trivial task in KRL to insert an image after all elements with an id of “bacon”.  It has been an order of magnitude more difficult task to find out what the contents of that element might be.

Introducing the query() operator.  Using jQuery-like syntax you can now query a page and inspect the elements themselves.

Technically, query works on strings or arrays of strings so the first task is getting the target html into a form that query can consume.

datasources

It has always been possible to point a dataset or datasource at a static URL and get the contents.  When the datasource fails to convert the text to a JSON structure, it returns the page contents as a string.  We now have the syntax to formalize the operation—we let the engine know that it doesn’t have to bother trying to convert the datasource content to JSON.

dataset

• Return the contents of a page or query as a string of HTML

Usage

	dataset fizz_data:HTML <- "http://www.htmldog.com/examples/tablelayout1.html"

datasource

• Return the contents of a page or query as a string of HTML

Usage

	datasource fozz_data:HTML <- "http://www.htmldog.com/examples/darwin.html"

The usage is the same as any dataset/source request; you just tag it with :HTML so the parser can skip the JSON step.

Query Operator

query

• Find all of the elements of the HTML string <source> that match the criteria established by <selector>

Usage

	<source>.query(<selector>)

           source : <string> | [<string1>, .. ,<stringN>]

           selector : <sel_string> | <sel_string1,sel_string2..sel_stringN> 

                                     | [<sel_string1>, .. , <sel_stringN>]

• Source could be a single string or an array of strings.  Query will apply the selection criteria to each string and return a single array of matching elements.
• Selector can be a jQuery style string, comma separated jQuery string, or an array of jQuery selector strings.

<selector>

query only supports a subset of the jQuery selectors for now:

• element
• #id
• .class
• [attr]
• [attr=value]

Given the following KRL variables:

global {

  dataset q_html:HTML <- "http://www.htmldog.com/examples/tablelayout1.html";

  dataset r_html:HTML <- “http://www.htmldog.com/examples/darwin.html”  

}

 

pre {
  in_str = <<
  th[colspan="2"]
>>;
  html_arr = [q_html,r_html];
  meta_str = <<td[style="background: #ddf;"]>>;
} 

element

• Matches all elements of a particular type

r_html.query("h1");
  matches <h1>
r_html.query(“caption,h1");
  matches either <caption> or <h1> 

#id

• Matches all elements with a specific id

q_html.query("#c_link");
  matches <… id=”c_link”>
q_html.query(“a#c_link");
  matches <a id=”c_link”>

.class

• Matches all elements with a specific class

q_html.query(".header");
  matches <… class=”header”>
q_html.query(“p.header");
  matches <p class=”header”>
q_html.query(“#c_link.header");
  matches <… id=”c_link” class=”header”>

[attr]

• Matches all elements with an attribute, even if attr=""

q_html.query("[style]");
  matches <… style=”…”>
q_html.query(“td[style]");
  matches <td style=”…”>

[attr=value]

• Matches all elements with an attribute set to a specific value

q_html.query("[align=center]");
  matches <… align=”center”>
q_html.query(“td[align=center]");
  matches <td align=”center”>
q_html.query(“[align=center][colspan=2]");
  matches <… align=”center” colspan=”2”>

Selectors support variable substitution.  Use this if you need to embed quotes

pre {
 in_str = << th[colspan="2"] >>;
 tbl_hdrs = q_html.query(in_str);
}
  matches <th colspan=”2”>
 
pre {
 in_str = << th[colspan="2"] >>;
 tbl_hdrs = q_html.query(#{in_str});
}
  matches <th colspan=”2”>

You can stack selectors

q_html.query("div#header span p[align=center]");
  matches <div id=”header”>… <span>… <p align=”center”>

If query’s operand is an array, the selector will be applied to each array element

  html_arr = [q_html,r_html];

  combo_arr = html_arr.query(“a”);
    matches <a> in either q_html or r_html   

 

You can join multiple selectors together as one string separated by commas or as an array of selector strings.

r_html.query(“caption,h1");
r_html.query([“caption”,”h1”]);
  matches either <caption> or <h1> (expressions are equal)

One final note: only the attribute equality selector is implemented.  jQuery has partial matches and negations in the selector syntax that won’t work with query.  If you return an empty array, it is either a sign that no HTML matched your selector or your selector syntax was wrong.

Not all JSON datasources are created equal.  The KRL JSON parser is fairly strict in requiring correct JSON.  We ran into one instance the other day where an API was using a single quote instead of double quotes.  The default behavior in KRL is to treat the datasource as a string if it can’t be parsed as JSON.

With the mis-behaving API, we ran into some trouble because we didn’t have any way to convert a string to JSON.  It would have been simple to replace the offending characters, but there was no where to go from there.

All of the mechanics for allowing the developer to do the JSON encoding were in place, but nothing was exposed.

The as operator coerces one type of object into another.  There are times when a string looks like a regexp, you want to concatenate numbers as strings or treat a number string (“123”) like a number (123).  KRL usually does a good job of figuring out what you want to do from the context, but sometimes you need to provide a hint.

Earlier, I showed how you could tail a persistent entity trail and turn it into an array.  It is now possible to turn a string into a JSON object that can be picked and passed unchanged between KRL and Javascript.  Of course, the string still needs to be able to pass our strict JSON parser, but it is possible now to take poorly formed JSON or a JSON fragment, massage it into valid JSON and create a JSON structure.

Syntax is straightforward:

Usage

	json_string.as("json")

The table of allowed coercions now looks like this:

Notice that the new operations are directional.  That is you can convert a string to JSON, but you can’t convert JSON to a string.  Similarly, you can convert a trail to an array, but not an array to a trail. Time permitting, I will add the opposite direction

Trails

During the recent Kynetx Code Run, quite a few teams latched on to using trails as a persistent entity (to pass information between rules).  Originally designed to provide a historical click path for rules, trails store data for your app per customer that will last longer than a single browsing session.

A trail resembles a stack.  You push your data onto a trail like you add a plate to those springy things at the buffet.  The existing plates shift down and your most recent plate is on the top.  Trails have a few distinct features:

• There is a limit to the number of items that you can keep in your stack.  If you are at the trail capacity and you push another item on, the oldest item falls out the bottom, lost to the impenetrable void.
• KRL provides a few functions to peek into the stack and pull out elements out of sequence
• Trails can only be changed in the postlude or callback sections of a rule.
• Trails have a timestamp associated with each entry

This is a sample of how we used trails to keep track of Facebook user ids:

// Get the user's recent Facebook posts
	facebook:get({"connection":"home"});   

// Make an array of the posts and Process each post individually 
	foreach facebook:get({"connection":"home"}).pick("$.data") setting (itm)   

// Get the FB id of the user that posted
	fid = itm.pick("$.from.id")   

// Decide whether or not you are going do something with this post
...   

// If we did use the post, add the user's id to a trail
	fired {
	  mark ent:sent_from with fid
	}  

Access to trails are through some special operations:

• current : takes the most recent entry on the trail and returns the value
• history : searches the trail for the first entry that matches an expression (regexp) and returns the value
• seen : like history, it searches the trail for an expression, but only returns true or false based on whether the value was detected*

seen is much more powerful than just that. Remember that each entity has a timestamp so you can build predicates based upon when the entity was added to the trail and whether entity ‘a’ was added before entity’ ‘b’.

Trails can be modified using these operations:

• clear : sets the persistent variable to null
• mark : pushes a new value on the top of the trail
• forget : searches for an expression in the trail and deletes it

One note on persistent entity variables: if you only want the scope of your variables to last for one ruleset execution, you must clear your variables at the beginning of your ruleset.

Arrays

One thing you can’t do with a trail is treat it like a normal KRL array.  It would be optimal if you could use foreach to parse over each element in a trail, but it was built before foreach was implemented and the persistent variable operations were not designed to support foreach.

Luckily, we have the as operator which can allow us to coerce the special trail array into a standard array which can be used by foreach and the other array operations like filter, head, tail, join, length, sort, and map. So the syntax becomes:

// Convert a persistent entity trail into a standard array
	ent:sent_from.as("array")   // Use as and trails to supply an array to foreach
	foreach ent:sent_from.as("array") setting user  

Easy 

Sets

So after building a list of Facebook uids pulled from the posts on a users home/news feed, I needed to grab the user photos to associate with said posts.  If you have a chatty friend on Facebook, they can have several posts on your wall at one time.  I have an array of FB ids, but there is a possibility that I might have duplicates and I DON’T want to query multiple times for the same picture.  I could probably work out a solution with seen on the trail side or a combination of filter and map on the array side, but if you have programmed in a language that supports set inclusion you resent the time spend building the functionality that can be represented by a simple in/has.

Since I can, I added set operations to KRL.

I did not implement the Cartesian product or the Power set.  I’ve found some programmatic applications which use the cross product, but it’s kind of like building a new house to replace your kitchen cabinets.  Maybe I will add it as soon as we add ordered pairs as a first class object.

Set Operations

• intersection
• union
• difference
• has
• once
• duplicates
• unique

I have tried to write into all the operations the ability to support scalars—single values.  I convert scalars to an array of length 1.  With the exception of has (which returns a boolean value), all set operators return an array.  The empty set [] is also supported.  Every array has the empty set as a subset.  Every array is a subset of itself.

Once, duplicates, and unique are convenience functions to address common programming tasks.  They are technically not set operations since arrays can have duplicate values, but you might notice that the arrays that these operations return are valid sets.

Given arrays (sets) A and B:

intersection (A ∩ B)

• The set of all objects that are a member of both A and B

Usage

	A.intersection(B)

Union (A ∪ B)

• The set of all objects that are a member of A or B (or both)

Usage

	A.union(B)

difference (A \ B)

• The set of all members of A that are not members of B

Usage

	A.difference(B)

has (B ⊆ A)

• B is a subset of A (or it could be read as A is a superset of B)

Usage

	A.has(B)

once

• Set of elements e that only appear 1 time in A 

Usage

	A.once()

duplicates

• Set of elements e that appear more than once in A 

Usage

	A.duplicates()

unique

• Set of (unique) elements e belonging to A

Usage

	A.unique()

These set operators behave just like any KRL operator so you can chain them to perform more complex operations.  This is how you could make your own symmetric difference operation:

symmetric difference

• Set of elements e belonging to either A or B, but not both

Usage

	(A.union(B)).difference(A.intersection(B))

Facebook has opened up access to most of their data through their Graph API. I bitch elsewhere about their decision to implement OAuth 2.0, but that should not detract from the fact that this new REST based interface is simple, straightforward, and a breeze to use.  The number of requests to parse a user’s feed is not insignificant compared to what is possible with FQL.

For our Kynetx Code Run, we have already done some pretty exciting things with the FB API. Previously, Kynetx has treated data sources as read-only operations.  With this release, we have introduced writes as a user action.  The format is similar to the usual get syntax, but it is an action so it needs to be called from the Action part of your rule (use it like you would notify or other actions like: after, before, float…)

Facebook calls their data the Social Graph.  I may use that term interchangeably with Facebook or the Graph API.

Skip to the examples

Create a Facebook application

The easy way that I did it was to just go to the Facebook Developer page. You will need to accept the FB terms and conditions and name your app.  It doesn’t matter what you name your app, but the name will be used by Facebook during the authorization process and for any content that you create through the API.

For our purposes, we are only interested in a couple bits of information; however, this does create a real app so you should spend some time filling in profiles, adding logos and generally tarting-up your app.

If you are familiar with OAuth, you know that you need an id and secret for the authorization process.  Conveniently for us, Facebook takes us straight to the required information

The Graph API uses the Application ID and Secret as the consumer_key and consumer_secret respectively.

It is important that you do the next step correctly!  Set your Facebook application’s Connect URL to http://cs.kobj.net:80/ruleset/fb_callback/ Facebook uses the Connect URL as an additional security measure to make sure developers are not up to something naughty.  As part of the OAuth process, KRL tells FB to send a token to a URL on our server (That’s how your user can log in and authorize through Facebook and never give out their username and password).  If your Connect URL and the URL that Kynetx provides don’t match, the authorization process will FAIL.

FACebook’s social graph

The Graph API exposes several objects from the social graph. From what I have read on the forums, there are a few objects that were available in the old API or FQL that are not available from the Graph API. I would recommend that you become familiar with what information is available and where to find it.  From the Facebook documentation, these are the objects that are available

• album
• event
• group
• link
• note
• page
• photo
• post
• status (message)
• user
• video

Create your Kynetx Application

For KRL, the process of integrating Facebook into your application is the same as the process we use for Twitter and Google.  The Twitter documentation is fairly extensive, so I will just hit the highlights.

Set up your facebook application information in the META section

key facebook {
	"consumer_key" : "892934849920029",
	"consumer_secret" : "1a3836d9d025f9e2738bb50976"
}

Create first rule to AUTHORIZE your user with Facebook

rule auth_app is active {
  select using ".*" setting ()
    if (not facebook:authorized())
      then
        facebook:authorize(["publish_stream","email","user_photos","read_stream",
          "user_notes","offline_access","user_about_me","user_notes",
          "user_photos","user_likes","user_online_presence"
        ])
        with opacity = 1 and
          sticky = true;
    fired {
      last;
    }
}

Before any FB request, KRL checks to see if the application has been authorized to access the current user’s information.  The auth_app rule explicitly runs that check for you.  If the user does not have a valid OAuth token, the OAuth authorization process is started.

Let’s examine the authorize command in more detail:

facebook:authorize(["publish_stream","email","user_photos",
  "read_stream","user_notes","offline_access","user_about_me",
  "user_notes","user_photos","user_likes","user_online_presence"
])

authorize takes as an argument an array of extended permissions.  They are separated into two major categories: Publishing and Data.  Data is further divided into User and Friends permissions.

Permissions

The full breakdown on permissions is found on Facebook.  Two permissions which are particularly relevant to KRL are publish_stream and offline_access. Without publish_stream permission, you won’t be able to post to your user or their friends accounts. Without offline_access, users would have to be logged into facebook for your queries to work. Here are the allowed values—any other values will just be ignored.

I suspect that some might be tempted to pile all the permissions into an authorize request. Aside from missing the point of writing context sensitive rules to fulfill specific needs, there is an Internet Explorer limitation which restricts the size of an HTTP query.  Just to keep you from piling on permissions, I’m not going to tell you what that limit is.

Despite the permissions the user has authorized, there is often public data which is available regardless of authorization. Do not be misled by a user’s public information into thinking that your app is authorized to request the information; ie: to allow other people to find you in a friend search, your first and last name has public permissions. So anyone who types the URL: https://graph.facebook.com/100001078761602/ will see the information that Kay Netticks has made public.

Connections

Connections are the way that facebook links objects together.  Most every object has connections to other objects.  This allows a group to have members, photos and a news feed.  A status message only has comments.  The user object has many, many connections—basically all the various stuff that you can see on your FB home page.

Not specifying a connection will return the profile information for the current object.

For convenience, there is a special function facebook:connections() which will tell you what are valid connections for a particular object.

IDs

Almost every object in the Social Graph has an id.  Getting information for a particular object is as easy as providing the correct id.  If you don’t specify an id, the currently authorized user is used as the default object.  Facebook has a special alias for the currently authorized user “me”.  KRL doesn’t require you to specify an id, but you may see request urls in your debug statements that use “me” so you should be aware of the meaning.

Paging

Several of the Social Graph queries can return multiple items.  Facebook provides some paging functions so you have some control over which and how many results are returned.

• limit
• offset
• since
• until

limit and offset both require integers for their values.  With limit, you can specify how many items you want to receive for one request. Offset allows you to skip any number of elements before you start returning results.

Since and until operate like a min and max date respectively.  They take a date as an argument, but that date can be formatted according to what the PHP strtotime function allows.  That means that ‘2010-04-01’ and ‘yesterday’ are both valid values.  The online PHP manual gives these as valid examples:

Valid formats for FBs since and until query parameters


"now”

"10 September 2000"

"+1 day”

"+1 week”

"+1 week 2 days 4 hours 2 seconds"

"next Thursday"

"last Monday”

fields

If you are only interested in select fields of a Social Graph object, you may use the fields argument to limit the response to only your desired fields. The argument takes the form:

• fields : <string> | <CSV string> | [<string1>,…,<stringN>]

A field name string, comma separated list of field names, or an array of field names are all valid values for fields.  For convenience, there is a special function facebook:fields() which will tell you what are valid fields for a particular object.

Queries

Query syntax should be familiar to KRL users.  There are 5 basic facebook queries which are available in KRL. Each can accept a hash as an argument, though facebook:search is the only function that requires an argument.  Not specifying an id, will perform the query in the context of the currently authorized user.

facebook:metadata()

• optional
• id :<string>

I debated whether I should provide access to the metadata query.  The metadata is designed to allow for introspection.  That’s kind of like telling you what know about what you know about an object.  It returns profile information, but it also returns preformatted url references to all of the connections that are valid for your object.  It’s a good idea, but when you have a token with your metadata request and are defaulting to the currently authorized user (“me”), the urls come pre-configured with the access token attached.  That may not make much difference for people who are writing pure javascript implementations of the Social Graph, but KRL is a MUCH more secure environment and I don’t like the idea of leaving that token lying around.

facebook:picture()

• optional
• id :<string>
• type : square | small | large

Most all objects have a picture or an icon associated with them. When you make a request for the picture, Facebook returns a redirect to the actual URL.  I follow that re-direct and return the actual URL for the picture.  This makes a picture call a little bit more complicated so bear that in mind if you are going to do large amounts of picture requests.  If it would be more useful, I could include something like a ‘nofollow’  tag so that I just return the location of the redirect, but that is currently not provided

facebook:search()

• required
• type : post | user | page | event | group | home
• q : <string>
• optional
• id :<string>

I haven’t found FB’s search functionality to be terribly robust. As I say to my kids, “You get what you get and you don’t throw a fuss”. If you find that you aren’t getting the results that you think you should, grab the url from the debug and enter it with your browser to see if it’s just bad indexing on FB’s part.

home is a special case where the search will be performed only on the currently authorized users news feed or whomever is specified by the optional id parameter

facebook:get()

• optional
• id :<string>
• connection: <string>
• type : <facebook object> 

If you would like to follow a connection; ie: get all the comments associated with a post, just provide a “connection” : <value>.  If you would like KRL to do some sanity checking, you can use the optional “type” : <object> syntax in conjunction with connection.  KRL will check to see if your connection is valid for that type of object.  If not, a warning will be printed to your console.

facebook:ids()

• required 
• ids :<string> | <CSV string> | [<string1..stringN>] | url

This method allows you to get profile information for multiple ids. <string> can be a username or the numeric id (it is not guaranteed that the user will have a username defined).

An optional usage is to provide a URL instead of an id.  You can use this to look up or check to see if there is an object already associated to a URL in Social Graph

POSTS

Facebook allows you to create new instances of a few of their Social Graph objects; ie: write a message on someone’s wall.  This requires the publish_stream permission from the user.  What information you supply is determined by the type of object that you are trying to create.

Here are the objects that you can publish via the Graph API:

Syntax for publishing is very simple, but remember that publishing is an action.  In this case, the action is not producing JavaScript, but posting data to the Graph API.

facebook:post()

• required 
• connection : <string>
• id: <string>
• optional
• <arg> : <string>

See table 2 for a list of what arguments are available for each connection.

*id is optional when the connection requires a profile id.  As in previous cases, if the id is not supplied, it will default to the id of the currently authorized user.  The following connections require a profile id: feed, notes, links, events, albums.  I would suggest always supplying the id for consistency.

The post method cross checks the connection type with the permitted arguments.  If an argument is not associated with the connection, it will just be ignored.

For convenience, there is a special function facebook:writes() which will tell you what are valid argument keywords for a particular connection.

Examples

facebook:metadata()

Request for metadata by id


facebook:metadata({“id” : “100001078761602”});
returns: JSON

Request for metadata for currently authorized user


facebook:metadata();
returns: JSON

Writable fields/allowed parameters for a Facebook object

facebook:writes(“feed”);
returns: Array

facebook:post()

Post a message to a user’s feed

facebook:post({"id" : "100001078761602",
	"connection" : "feed",
	"message" : "Contactless not, swipe again",
	"picture" : "https://kynetx-images.s3.amazonaws.com/KynetxLogo273.png",
	"link" : "http://www.kynetx.com",
	"name" : "Things I like",
	"description" : "I am integrated into Facebook!" });

“like” a post

facebook:post({"id" : "641349049_124260134272950",
	"connection" : "likes"});

facebook:ids()

Search for these ids and return their profile data

facebook:ids(“ids” : [“100001078761602”,”116360591732083”]);
returns: JSON

Search the Facebook Social Graph to see if a URL has an ID

facebook:ids(“ids” : “http://www.kynetx.com”);
returns: JSON

facebook:get()

Request specific user’s feed


facebook:get({“id” : “100001078761602”, ”connection” : “feed”});
returns: JSON

Request default user’s albums


facebook:get({”connection” : “albums”});
returns: JSON

Request default user’s home with optional “type” syntax


facebook:get({”connection” : “home”, ”type” : “user”});
returns: JSON

Request an object by id


facebook:get({”id” : “511048495_446064733495”});
returns: JSON

Request all the messages posted to a specific object (link)


facebook:get({“id” : “511048495_446064733495”, ”connection” : “comments”});
returns: JSON

facebook:search()

Search the Social Graph for a page


facebook:search({”type” : “page”, ”q” : “Lehi, Utah”});
returns: JSON

Search the Social Graph for a user


facebook:search({”type” : “user”, ”q” : “Steve Fulling”});
returns: JSON

facebook:picture()

Get the large picture associated with the currently authorized user

facebook:picture({“type” : “large”});
returns: url (string)

Get the picture for an object

facebook:picture({“id” : “116360591732083”});
returns: url (string)

Request for object picture by id and type


facebook:picture({“id” : “100001078761602”, ”type” : “small”});
returns: url (string)

facebook:connections()

Allowed connections for a Facebook object

facebook:connections(“page”);
returns: Array

facebook:fields()

Allowed fields for a Facebook object

facebook:fields(“user”);
returns: Array

facebook:authorize()

Authorization Action with desired Facebook permissions

facebook:authorize(["publish_stream","email","user_photos","read_stream"]);

facebook:authorized()

Authorization predicate

facebook:authorized();
returns: true or false

limit

Search the Social Graph for a group and limit results to 3


facebook:search({"type" : "post",
        "q" : "bacon",
        "limit" : 3});
returns: JSON

offset

Search the Social Graph for a user and start with the 20th object


facebook:search({"type" : "user",
        "q" : "John Smith",
        "offset" : 20});
returns: JSON

until

Search the Social Graph for users named  who signed up before 2010-05-18


facebook:search({"connection" : "feed",
        "q" : "John Smith",
        "until" : "2010-05-08"});
returns: JSON

since

Search default user’s news for posts made within last day


facebook:get({"connection" : "feed",
        "since" : "yesterday”});
returns: JSON

fields

Get specific fields for the default user


facebook:get({“fields” : [“last_name”,”email”]});
returns: JSON

The other day, I stumbled onto a philosophy that could make the world a better(TM) place.

Kynetx is a pretty cool place.  Every Friday we host an event that we call, “Free Lunch Fridays”.  All FOK or potential FOK are welcome to come to the Kynetx World Headquarters for lunch and a chance to talk with the Kynetx developers or biz dev team (or you could just eat).

One recent Friday morning, someone in the office asked what we were going to have for lunch.  After some discussion about choosing a sandwich franchise (not that one), it was suggested that we order J Dawgs.  At Kynetx, we are great fans of the J Dawgs.  Menu options are streamlined:

The previous day, I had spent all day trying to decipher the new API of a large social media network (Yes, that one).  It had not gone well.  Documentation was just shy of half-assed (and not the good half); the errors reported by the API were not helpful; I was reduced to blind guesses at what the implementation required.  I was not in a patient and forgiving mood.

Maybe I really wanted a hot dog that day, but when my buddy speculated that it was too late in the morning for J Dawgs to put together a large order, I loudly suggested that, “How about we let JDawgs decide [whether or not they could take our order]?!”

Even though I felt bad about putting too much vinegar into the comment, after J Dawgs was contacted and didn’t have a problem with timing or size, I figured that there was a Life Lesson to be learned from the incident.

Stop making assumptions about what someone else wants

Corollary: It’s okay to ask

Don’t be afraid to ask for something that you want.

Most people subscribe to some form of customer service philosophy.  Satisfied customers are more valuable than hostages.  If you remove the commercial aspect (because that can get creepy), we can extend this concept to just about any relationship.  One of the most frustrating aspects of working on product development is assigning value.  Too many times have I heard, “To sell this we need x” or “the user wants y”, but, upon further questioning, there is no data to support the assertion, just opinions and suppositions.

I had a garage sale over the weekend.  We put prices on all of the items, but at the end of the day as we got more and more tired and prospect of giving the stuff that didn’t sell to charity grew, we were much more open to negotiations.  What I wouldn’t negotiate on was the collection of paperbacks that I had in the sale.  I knew that a local used book store would offer me a dollar credit per title. the  In GTY terms, I had a BATNA

In Kynetx terms, value has context.  Asking for something helps both parties establish what is relevant to the deal.

Corollary: It’s okay to say no

If you get a request that doesn’t pass your internal cost/benefit, risk analysis, it’s okay to turn it down.

Both parties have to understand that a ‘no’ is not a personal attack.  Some people are better at saying no than others; they can make you feel like they are doing you a favor by turning you down. There is a business book out called, “Getting to Yes” (I haven’t read it. Not because it’s a bad book, but because I have a general prejudice about business books) which implies that there is plenty of ‘no’ in the meantime.

This morning my boss articulated the unifying principle that makes this philosophy work:

Base line: Don’t be a jackass

I don’t believe in zero-sum attitudes.  I had a boss once that regards negotiations as a scorched earth policy.  Relationships were ruined, deals soured because he wanted to squeeze every penny out of the guy across the table.

Some people are pleasers.  They might still buy into the meme that the “customer is always right”.  If that was true, everyone would have real broadband for $14.99, an In-and-Out Burger on every corner, and Michael Bolton would be prohibited from appearing in public.

If you aren’t interested in both parties finding a mutually acceptable solution, you might be a jackass (with apologies to Mssr. Foxworthy).

School House Rock! taught me that ‘3’ is a magic number.  It takes 3 legs to make a tripod. Just so, without all 3 of these principles, Let JDawgs Decide will fail.

In my alternate universe as the widely acknowledged godfather of KRL datasource integrations, I have the great privilege to be integrating the Facebook graph API into KRL.

In a ploy to increase billable hours for developers world-wide, Facebook has chosen the draft protocol OAuth 2.0 as the guardian of all things Farmville.  In a related effort to increase impression revenues for DevX and any number of FB developer forums, they have provided potty-poor documentation on integrating token based authentication.

OAuth 2.0 provides for 3 types of User Delegation Flow. If you aren’t familiar with OAuth, user delegation flow is a way of giving a third party access to your Facebook information without having to give up your password.  It also allows the user to revoke said 3rd party’s access at any time without requiring a password change.

FB does this by issuing a token (think hall pass) to an application that has to be provided in all requests for protected information. 

OAuth Token Exchange Process

I didn’t want to explain this, but I got further down into the ‘splanation I realized that I was using more and more mumbo-jumbo, blah, blah, blah.  So here is my explanation (lies kept to a minimum, but greatly oversimplified to the point of ridiculousness)

We have three parties:

• Facebook (FB)
• Facebook User (User)
• Not Facebook Application (app)

Our app would like to know to which Facebook groups the User belongs.  The User is okay sharing that group information with our app, but doesn’t want to give app access to User’s friend list and definitely not any login information.  FB’s plans for world domination include addicting everyone to their platform and would love to give access to the data, but they don’t want to anger their customers (again).

Here is how the magic happens:

1. app creates an account with FB and FB gives app an app.id and app.password
2. User installs, runs app (app doesn’t (have to) know anything about the User, just that the User is interested in sharing their x,y,z information from FB)
3. app tells FB, I have a User and they would like to share their group information
4. FB asks User, “Who are you?” (login or cookie)
5. FB asks User will you allow app to know x,y,z
6. User tells FB yes or no
7. if yes, FB tells app (callback) “Here is a token that you must give me every time you ask about User’s x,y,z”

DBaJ: The user can revoke the token at any time.  Doesn’t have to ask you.  Just go to the (FB application) permissions page and delete.  As well,  if you tell the User that you are going to use x,y,z, but you tell FB that you really want a,b,x,y,z, FB tells the User that you are asking for a,b,x,y,z.

I’m only interested in 2 of the flows:

• User Delegation Flow
• Web Server Flow

User-Agent Flow

In the user-agent scenario, the browser-based application communicates directly with Facebook.  This is important to understand because the access_token response is returned to the application as a fragment.  A fragment is anything after the ‘#’ in a url.  I don’t consider this very secure, but it does keep that token from showing up in any server logs. 

I’m not a Javascript guy, so I don’t know how secure Javascript is about keeping a rogue script from discovering the token.  If a bad guy gets the token, they have complete, unfettered access to the user information as if they were you.

Web Server Flow

In this scenario, the developer has a secure server that the Facebook server can ‘callback’ the app server with the token information.  In the draft spec, this is shown as a POST request (which rules out cross-domain javascript), but the Facebook documentation shows a GET implementation.

POST /token HTTP/1.1

     Host: server.example.com

     Content-Type: application/x-www-form-urlencoded


     type=web_server&client_id=s6BhdRkqt3&

     client_secret=gX1fBat3bV&code=i1WsRn1uB1&

     redirect_uri=https%3A%2F%2Fclient%2Eexample%2Ecom%2Fcb

I tried the web server flow first with a GET request per the FB documentation, but kept getting a verification error.

I tried out the user-agent flow, before I found out that clients don’t pass fragments along to the server.

–> Update 13 May 2010   Facebook’s Graph API and Web Server Flow

There are a few posts out there that recommend using type=client_cred; ignore those because it is just a red herring.  client_cred is a token type for the Client Credentials Flow.   This is for an app request to FB, not on the behalf of user.  Facebook uses this for their analytics interface, it is for access to app information.  The token will seem to work, but you will get the same results if you don’t supply any token when you are making a user request.  You just get the public information.  If you can’t make a request using the https://graph.facebook.com/me syntax, you don’t have a valid user access token.

Making web_server flow work

If you don’t supply the type=<flow type> parameter, it appears that Facebook defaults to the web_server flow.  The draft has this information about the web server flow parameters:

redirect_uri
         REQUIRED unless a redirection URI has been established between
         the client and authorization server via other means.  An
         absolute URI to which the authorization server will redirect
         the user-agent to when the end user authorization step is
         completed.  The authorization server MAY require the client to
         pre-register their redirection URI.  The redirection URI MUST
         NOT includes a query component as defined by [RFC3986] section
         3 if the "state" parameter is present.
state
         OPTIONAL.  An opaque value used by the client to maintain state
         between the request and callback.  The authorization server
         includes this value when redirecting the user-agent back to the
         client.

It took me a while and reading through a lot of information on Ben Beddinton’s blog and the Graph API forum, but I noticed that all of the examples of working token requests had simple redirect_uri references.

redirect_uri=http%3A%2F%2Fbenbiddington.wordpress.com

redirect_uri=http://www.example.com/callback

redirect_uri=http://localhost:8888/auth/fb

 

Because we can execute Kynetx apps on any page, we need to be able to send additional information with the callback (redirect_uri) so we can redirect the user back to the correct page.  The draft says that this is okay as long as the state parameter is not set.  Unfortunately, the FB implementation sets the state field by default or just can’t handle queries in a callback.

As soon as I removed the query part of the URI, I was able to get a valid token.  I still needed to pass information back to my server, so I had to re-write the callback URL to include the needed parameters as part of the URL path (Like REST).

I now have full access to FB’s graph API, and am continuing with the predicate implementation so KRL developers will soon have access to FB GET and search functionalities from within KRL.

This file is found at the OData demo database: http://services.odata.org/OData/OData.svc/$metadata The complete XML is at the end of the article. If you’re an XML whiz, you won’t have any trouble with this file.

Hey, I like XML.  You can pack a lot of information into XML.  Most of the guys at work hate XML—go figure. Let’s look at some highlights.

This snip defines the collections available

  1: <EntityContainer Name="DemoService" m:IsDefaultEntityContainer="true"> 
  2:   <EntitySet Name="Products" EntityType="ODataDemo.Product" /> 
  3:   <EntitySet Name="Categories" EntityType="ODataDemo.Category" /> 
  4:   <EntitySet Name="Suppliers" EntityType="ODataDemo.Supplier" />
  5: ... 
  6: </EntityContainer>

The ellipses means that I’m hiding some stuff from you.  Have I mentioned that I am a lying, liar?

Line 2 says:  I have a collection named “Products”.  It is made up of “Product”s.  “Product” is defined elsewhere in the file.

I’m not sure why OData stuck with the Entity-ish syntax.  For data lube, this sure seems like sand in the vaseline.  Did I mention that Java is a cuss-word at work?  Stinking punks have no appreciation… anyways, here is a table from the OData site that reconciles the names used by OData with the XML tags used in the meta document.

  1: <EntityType Name="Product"> 
  2:   <Key> 
  3:      <PropertyRef Name="ID" /> 
  4:   </Key> 
  5:   <Property Name="ID" Type="Edm.Int32" Nullable="false" /> 
  6:   <Property Name="Name" Type="Edm.String" Nullable="true" m:FC_TargetPath="SyndicationTitle" m:FC_ContentKind="text" m:FC_KeepInContent="false" /> 
  7:   <Property Name="Description" Type="Edm.String" Nullable="true" m:FC_TargetPath="SyndicationSummary" m:FC_ContentKind="text" m:FC_KeepInContent="false" /> 
  8:   <Property Name="ReleaseDate" Type="Edm.DateTime" Nullable="false" /> 
  9:   <Property Name="DiscontinuedDate" Type="Edm.DateTime" Nullable="true" /> 
 10:   <Property Name="Rating" Type="Edm.Int32" Nullable="false" /> 
 11:   <Property Name="Price" Type="Edm.Decimal" Nullable="false" /> 
 12:   <NavigationProperty Name="Category" Relationship="ODataDemo.Product_Category_Category_Products" FromRole="Product_Category" ToRole="Category_Products" /> 
 13:   <NavigationProperty Name="Supplier" Relationship="ODataDemo.Product_Supplier_Supplier_Products" FromRole="Product_Supplier" ToRole="Supplier_Products" /> 
 14: </EntityType> 

Just like bears eat beets, Product has properties.  The XML tells you what the name is, whether it’s a string or int blah, blah blah.  The important thing  is that you see that every entity has a key and that key is one of the properties.  In this case, “ID”.  If you want to pull a specific Entry out of a collection, you need to know the key.

Another thing to note is the NavigationProperty.  Consider it a link or a relationship to another collection.  In this case, Product has a link to Category and a link to Supplier. So, every product has a Category and a Supplier, but those are collections in their own sense, and not something simple like Price.

If you have something that is more complex than Price or Name, but they just aren’t important enough to be a collection (or there isn’t a key defined for it) you can put it in a complex type.  In our example, every supplier has an address.  So, Supplier has the property “Address”.

  1: <Property Name="Address" Type="ODataDemo.Address" Nullable="false" /> 

and this is what an address looks like:

  1: <ComplexType Name="Address"> 
  2:   <Property Name="Street" Type="Edm.String" Nullable="true" /> 
  3:   <Property Name="City" Type="Edm.String" Nullable="true" /> 
  4:   <Property Name="State" Type="Edm.String" Nullable="true" /> 
  5:   <Property Name="ZipCode" Type="Edm.String" Nullable="true" /> 
  6:   <Property Name="Country" Type="Edm.String" Nullable="true" /> 
  7: </ComplexType>

The nice thing about this is that if you want to know what the supplier’s name is but don’t really care what the address is, you don’t have to retrieve all that information when you get your supplier.  OData will give you a link to go get it if you change your mind.  Conversely, if you want the supplier’s name and address, you just tell your query to $expand address and you get the full address without having to make a second OData call.

One last thing.  A Service Operation is an OData function call.  Service operations can accept key=value parameters.

  1: <FunctionImport Name="GetProductsByRating" EntitySet="Products" ReturnType="Collection(ODataDemo.Product)" m:HttpMethod="GET"> 
  2:   <Parameter Name="rating" Type="Edm.Int32" Mode="In" /> 
  3: </FunctionImport>

the whole $metadata file

  1: <edmx:Edmx Version="1.0" xmlns:edmx="http://schemas.microsoft.com/ado/2007/06/edmx"> 
  2:   <edmx:DataServices xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" m:DataServiceVersion="2.0"> 
  3:     <Schema Namespace="ODataDemo" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://schemas.microsoft.com/ado/2007/05/edm"> 
  4:       <EntityType Name="Product"> 
  5:         <Key> 
  6:           <PropertyRef Name="ID" /> 
  7:         </Key> 
  8:         <Property Name="ID" Type="Edm.Int32" Nullable="false" /> 
  9:         <Property Name="Name" Type="Edm.String" Nullable="true" m:FC_TargetPath="SyndicationTitle" m:FC_ContentKind="text" m:FC_KeepInContent="false" /> 
 10:         <Property Name="Description" Type="Edm.String" Nullable="true" m:FC_TargetPath="SyndicationSummary" m:FC_ContentKind="text" m:FC_KeepInContent="false" /> 
 11:         <Property Name="ReleaseDate" Type="Edm.DateTime" Nullable="false" /> 
 12:         <Property Name="DiscontinuedDate" Type="Edm.DateTime" Nullable="true" /> 
 13:         <Property Name="Rating" Type="Edm.Int32" Nullable="false" /> 
 14:         <Property Name="Price" Type="Edm.Decimal" Nullable="false" /> 
 15:         <NavigationProperty Name="Category" Relationship="ODataDemo.Product_Category_Category_Products" FromRole="Product_Category" ToRole="Category_Products" /> 
 16:         <NavigationProperty Name="Supplier" Relationship="ODataDemo.Product_Supplier_Supplier_Products" FromRole="Product_Supplier" ToRole="Supplier_Products" /> 
 17:       </EntityType> 
 18:       <EntityType Name="Category"> 
 19:         <Key> 
 20:           <PropertyRef Name="ID" /> 
 21:         </Key> 
 22:         <Property Name="ID" Type="Edm.Int32" Nullable="false" /> 
 23:         <Property Name="Name" Type="Edm.String" Nullable="true" m:FC_TargetPath="SyndicationTitle" m:FC_ContentKind="text" m:FC_KeepInContent="true" /> 
 24:         <NavigationProperty Name="Products" Relationship="ODataDemo.Product_Category_Category_Products" FromRole="Category_Products" ToRole="Product_Category" /> 
 25:       </EntityType> 
 26:       <EntityType Name="Supplier"> 
 27:         <Key> 
 28:           <PropertyRef Name="ID" /> 
 29:         </Key> 
 30:         <Property Name="ID" Type="Edm.Int32" Nullable="false" /> 
 31:         <Property Name="Name" Type="Edm.String" Nullable="true" m:FC_TargetPath="SyndicationTitle" m:FC_ContentKind="text" m:FC_KeepInContent="true" /> 
 32:         <Property Name="Address" Type="ODataDemo.Address" Nullable="false" /> 
 33:         <Property Name="Concurrency" Type="Edm.Int32" Nullable="false" ConcurrencyMode="Fixed" /> 
 34:         <NavigationProperty Name="Products" Relationship="ODataDemo.Product_Supplier_Supplier_Products" FromRole="Supplier_Products" ToRole="Product_Supplier" /> 
 35:       </EntityType> 
 36:       <ComplexType Name="Address"> 
 37:         <Property Name="Street" Type="Edm.String" Nullable="true" /> 
 38:         <Property Name="City" Type="Edm.String" Nullable="true" /> 
 39:         <Property Name="State" Type="Edm.String" Nullable="true" /> 
 40:         <Property Name="ZipCode" Type="Edm.String" Nullable="true" /> 
 41:         <Property Name="Country" Type="Edm.String" Nullable="true" /> 
 42:       </ComplexType> 
 43:       <Association Name="Product_Category_Category_Products"> 
 44:         <End Role="Product_Category" Type="ODataDemo.Product" Multiplicity="*" /> 
 45:         <End Role="Category_Products" Type="ODataDemo.Category" Multiplicity="0..1" /> 
 46:       </Association> 
 47:       <Association Name="Product_Supplier_Supplier_Products"> 
 48:         <End Role="Product_Supplier" Type="ODataDemo.Product" Multiplicity="*" /> 
 49:         <End Role="Supplier_Products" Type="ODataDemo.Supplier" Multiplicity="0..1" /> 
 50:       </Association> 
 51:       <EntityContainer Name="DemoService" m:IsDefaultEntityContainer="true"> 
 52:         <EntitySet Name="Products" EntityType="ODataDemo.Product" /> 
 53:         <EntitySet Name="Categories" EntityType="ODataDemo.Category" /> 
 54:         <EntitySet Name="Suppliers" EntityType="ODataDemo.Supplier" /> 
 55:         <AssociationSet Name="Products_Category_Categories" Association="ODataDemo.Product_Category_Category_Products"> 
 56:           <End Role="Product_Category" EntitySet="Products" /> 
 57:           <End Role="Category_Products" EntitySet="Categories" /> 
 58:         </AssociationSet> 
 59:         <AssociationSet Name="Products_Supplier_Suppliers" Association="ODataDemo.Product_Supplier_Supplier_Products"> 
 60:           <End Role="Product_Supplier" EntitySet="Products" /> 
 61:           <End Role="Supplier_Products" EntitySet="Suppliers" /> 
 62:         </AssociationSet> 
 63:         <FunctionImport Name="GetProductsByRating" EntitySet="Products" ReturnType="Collection(ODataDemo.Product)" m:HttpMethod="GET"> 
 64:           <Parameter Name="rating" Type="Edm.Int32" Mode="In" /> 
 65:         </FunctionImport> 
 66:       </EntityContainer> 
 67:     </Schema> 
 68:   </edmx:DataServices> 
 69: </edmx:Edmx>

Jon Udell, the James Brown of social media, famously said that OData is “grease to cut data friction”.

OData and GData are very similar.  They both aim to provide a rich data interface over the HTTP protocol (RESTish) using Atom as the data medium.  Rather than hash up an explanation with my lying, liar-ness, once again here is a link to the resources I used to figger things out:

• OData Developer Documentation

In OData, information is grouped into Collections of Entries (Atom-ish).  Atom is extended to provide primitive and complex data types and relationships between Collections (This magic happens in a metadata document).  A big advantage of OData is that the metadata can be exposed to define an Entity Data Model.

Managing Expectations

As with our other integrations, the primary goal has been to provide access to new sources of information (and context).  Creating, Updating, and Deleting are beyond the scope of this project.

OData 101 98

Remember that OData is not the database.  OData is the HTTP layer that mediates between web requests and the source data.  It is a data service that says, “Hey, this is the information that I know.  Don’t worry about how I know, I will take care all those messy details for you!”  In that, I just realized that OData is a (targeted) permutation of what KRL provides.

I think it’s an interesting choice of words in the O’documentation: “OData services may provide two types of metadata documents to describe themselves” (I added the bold face, blah, blah, for the usual reasons).  The ability to auto-discover the complete data model just by knowing the source URL seems like the whole point…  We’ll just pretend that everyone makes metadocuments available.  If they don’t, I’ve decided that they are idiots and ‘shouldn’t be trusted’.

Here is the service document from odata.orgs test server:

  1: <service xml:base="http://services.odata.org/OData/OData.svc/" xmlns:atom=".../2005/Atom">
  2:   <workspace>
  3:     <atom:title>Default</atom:title>
  4:     <collection href="Products">
  5:       <atom:title>Products</atom:title>
  6:     </collection>
  7:     <collection href="Categories">
  8:       <atom:title>Categories</atom:title>
 10:     </collection>
 11:     <collection href="Suppliers">
 12:       <atom:title>Suppliers</atom:title>
 13:     </collection>
 14:   </workspace>
 15: </service>

It shows that there are 3 collections that you can access: Products, Categories, Suppliers.  I have created a function so that you can grab this information.  It might not be production useful, but I figure that it might be helpful while you are developing.  In OData, case matters!

  1: rule get_entity_sets is active {
  2:   select using ".*" setting ()
  3:   pre {
  4:     otmp = odata:entity_sets("http://services.odata.org/OData/OData.svc");
  5:   }
  6: 
  7:   notify("Available collections", "#{otmp}")
  8:     with
  9:       sticky = true;
 10: }

This function will return an array of the collection names.

OData queries resemble a mix of REST and LDAP.  I’m not going to take any time explaining how the actual URLs are constructed.  It should be sufficient to know that I have tried to maintain the structure and ordering of the standard, but have made changes to accommodate the function/predicate nature of KRL.

You know, if you are really offended by my adaptation, you can always treat the OData call as a KRL datasource and build the URL with your own two, ungrateful, little hands.

KRL OData Functions

entity_sets

odata:entity_sets(<service_root>)

service_root: url to OData service

returns an array of the Collection names

metadata

odata:metadata(<service_root>)

service_root: url to OData service

returns a json translation of the XML metadata document

service_operation

odata:service_operation(<service_root>,<function_name>,{qparam1:qvalue1|,qparam2:qvalue2|..})

service_root: url to OData service

  function_name: service operation name (from $metadata)

  qparam:qvalue: optional hash of key:value pairs

    qparam: query parameter name

    qvalue: query parameter value

return varies per implementation

get

The ruleset at the end of this article has multiple rules.  Each rule has an example of how to use the odata:get function.

odata:get(<service_root>,<resource_path>,<query_options>)

service_root: url to OData service 
  resource_path: arg1 | [arg1,arg2,..argn]

  query_options: arg | { qparam1:qvalue1, qparam2:qvalue2 …}

resource path

Resource Path	Description
“Categories”	Identifies all Categories Collection
{“Categories”:1}	Identifies a single Category Entry with key value 1
[{“Categories”:1},”Name”]	Identifies the Name property of the Categories Entry with key value 1
[{“Categories”:1},”Products”]	Identifies the collection of Products associated with Category Entry with key value 1
[{“Categories”:1},{“Products”:1},”Supplier”,”Address”,”City”]	Identifies the City of the Supplier for Product 1 which is associated with Category 1

query options 

System Query	Example	Description
Orderby	“$orderby”:”Rating”	return in ascending order by property Rating
	“$orderby”:”Rating asc”	Same as above
	“$orderby”:”Rating,Category/Name desc”	Same as above AND subsequently sorted by the Name property of the Category property in descending order
Top	“$top”:5	Return the first 5 entries
Skip	“$skip”:2	return entries starting with the 3rd entry
Filter	“$filter”:”<filter syntax>”	See the filter syntax table for details*
Expand	“$expand:”Products”	As mentioned before, relationships are not expanded by default.  A link is provided to the data.  If you would like to list all of the Categories with each of the Products:      odata:get(<service_root>,”Categories”,{“$expand”:”Products”})
Format	“$format”:”Xml”	This will override the default behavior of returning a JSON structure.  This will have the effect of returning a string of XML that is not parsed by the KRL engine
Select	“$select”:”Price,Name”	Return on the the Price and Name properties
	“$select”:”Name,Category”	Returns just the Name and Catalog properties—Catalog is a reference to another collection so just a link is returned
	“$select”:”*”	return all properties
Inlinecount	“$inlinecount”:”allpages”	Return the total entries identified by the resource and $filter options and the entries
Count	“$count”	Returns just a scalar count of entries identified by the resource and $filter options
Value	“$value”	Returns the text value of a property query

filter syntax*

Operator	Description	Example
eq	Equal	“$filter”:”Address/City" eq ’Redmond’”
ne	Not Equal	“$filter”:”Address/City" ne ’London’”
gt	Greater than	“$filter”:”Price gt 20”
ge	Greater than or equal	“$filter”:”Price ge 10”
lt	Less than	“$filter”:”Price lt 20”
le	Less than or equal	“$filter”:”Price le 100”
and	Logical and	“$filter”:”Price le 200 and Price gt 3.5”
or	Logical or	“$filter”:”Price le 3.5 or Price gt 200”
not	Logical negation	“$filter”:”not endswidth(Description,’milk’)”
add	Addition	“$filter”:”Price add 5 gt 10”
sub	Subtraction	“$filter”:”Price sub 5 gt 10”
mul	Multiplication	“$filter”:”Price mul 2 gt 2000”
div	Division	“$filter”:”Price div 2 gt 4”
mod	Modulo	“$filter”:”Price mod 2 eq 0”
()	Precedence grouping	“$filter”:”(Price sub 5) gt 10”

Filter syntax (Strings, date, math, type)

There are a number of string, date, math, and type functions that are defined and should work, but I have not officially tested them.  You would need to modify the original syntax to fit the changes for KRL.  So, until I get a chance to expand testing (or someone bitches to devex about things not working) watch out for lions, tigers and bears.

Get’O’Rama

Here are some sample gets against the OData demo server (service_root = “http://services.odata.org/OData/OData.svc”) that you can use as a reference.

odata:get(service_root,"Products")

What: gets all of the entries from the Products collection

odata:get(service_root,["Products","$count"])

What: get the number of entries from the Products collection

odata:get(service_root,{"Products":2})

What: get the product with id=2

odata:get(service_root,[{"Products":2},"$links","Supplier"])

What: get the Supplier link for the product with id=2

odata:get(service_root,[{"Products":2},"Supplier","Address","City"])

What: get the Supplier’s city for the product with id=2

odata:get(service_root,[{"Products":2},"Supplier","Address","City",”$value”])

What: get the text value of the Supplier’s city for the product with id=2

odata:get(service_root,[{"Categories":1},"Products"])

What: return all the Products that belong to the Category with id=1

odata:get(service_root,[{"Categories":1,{"Products":1}])

What: get the product with id=1 from the Category with id=1

odata:get(service_root,[{"Categories":1},{"Products":1},"Name"])

What: get just the Name of the product with id=1 from the Category with id=1

odata:get(service_root,"Products",{"$expand":"Supplier"})

What: get all of the entries from the Products collection and fetch the values for the Supplier property as well

service_root="http://odata.netflix.com/Catalog/";
odata:get(service_root,"Titles",{"$filter":
     "Type eq 'Movie' and Instant/Available eq true and
     (Rating eq 'G' or Rating eq 'PG-13' or Rating eq 'PG')",
     "$top":2,
     "$orderby":"AverageRating desc",
     "$select":["Name","Synopsis","AverageRating"]})

What: From the Netflix OData catalog, get the Name, Synopsis, and AverageRating for the first 2 entries of instantly available movies sorted by Netflix rating, sorted Highest to Lowest by rating

Sample Ruleset

  1: ruleset a144x19 {
  2:     meta {
  3:         name "odata"
  4:         author "meh"
  5:         description <<
  6: odata tutorial app     
  7: >>
  8:         logging on    
  9:     }
 10:     dispatch {
 11:         domain "kynetx.com"
 12:         domain "odata.org"
 13:     }
 14:     global {
 15:         service_root = "http://services.odata.org/OData/OData.svc";    }
 16:     rule get_entity_sets is active {
 17:         select using ".*" setting ()
 18: 
 19:         pre {
 20:             otmp = odata:entity_sets(service_root);
 21:         }
 22:             notify("Available collections", "#{otmp}")
 23:               with
 24:                 sticky = true;
 25:     }
 26:     rule get_metadata is active {
 27:         select using ".*" setting ()
 28: 
 29:         pre {
 30:             md = odata:metadata(service_root);
 31:             name = md.pick("$..EntityContainer.@Name");
 32:         }
 33:             notify("metadata", name)
 34:               with
 35:                 sticky = true;
 36:     }
 37:     rule get_so is active {
 38:         select using ".*" setting ()
 39: 
 40:         pre {
 41:             sod = odata:service_operation(service_root, "GetProductsByRating", {"rating" : "3", "$expand" : "Supplier", "$inlinecount" : "allpages"});
 42:             pname = sod.pick("$..results[0].Name");
 43:             pprice = sod.pick("$..results[0].Price");
 44:             pdesc = sod.pick("$..results[0].Description");
 45:             blob = << 
 46:              <div>Product: #{pname}<br />
 47:                 Price: $#{pprice} <br />
 48:                Description: #{pdesc}  </div>  
 49:  >>;
 50:         }
 51:             notify("Service Operation", blob)
 52:               with
 53:                 sticky = true;
 54:     }
 55:     rule get_prod_count is active {
 56:         select using ".*" setting ()
 57: 
 58:         pre {
 59:             count = odata:get(service_root, ["Products", "$count"]);
 60:         }
 61:             notify("Number of Products", count)
 62:               with
 63:                 sticky = true;
 64:     }
 65:     rule get_linkage is active {
 66:         select using ".*" setting ()
 67: 
 68:         pre {
 69:             product = odata:get(service_root, [{"Products" : 2}, "$links", "Supplier"]);
 70:             murl = product.pick("$..uri");
 71:         }
 72:             notify("Supplier linked to Product(2)", murl)
 73:               with
 74:                 sticky = true;
 75:     }
 76:     rule get_supplier_city_value is active {
 77:         select using ".*" setting ()
 78: 
 79:         pre {
 80:             city = odata:get(service_root, [{"Products" : 2}, "Supplier", "Address", "City", "$value"]);
 81:         }
 82:             notify("Supplier city for  Product(2)", city)
 83:               with
 84:                 sticky = true;
 85:     }
 86:     rule get_prod_by_category is active {
 87:         select using ".*" setting ()
 88: 
 89:         pre {
 90:             product = odata:get(service_root, [{"Categories" : 1}, {"Products" : 1}]);
 91:             pname = product.pick("$..d.Name");
 92:             pprice = product.pick("$..d.Price");
 93:             pdesc = product.pick("$..d.Description");
 94:             blob = << 
 95:              <div>Product: #{pname}<br />
 96:             Price: $#{pprice} <br />      
 97:               Description: #{pdesc} </div>   
 98:  >>;
 99:         }
100:             notify("Categories(1) => Products(1)", blob)
101:               with
102:                 sticky = true;
103:     }
104: 
105: }

Everybody but Rupert Murdock loves Google.  And China.  And this guy. Here at Kynetx we love Google.  One of our engineers actually named his daughter Anne DeRoy.

Many of Google’s apis use the Google Data Protocol as the base for accessing their services.  GData uses a RESTish structure for calls, AtomPub (JSON support is nascent) for the format, and OAuth for security.

When KRL added support for XML, many feeds that didn’t require security were made available if you just treated the feeds as a datasource.  Unfortunately, all the cool context and information remained protected behind Google’s AuthSecure/OAuth layers.

When Phil cracked the OAuth dance with his Twitter implementation, KRL now had all of the elements to provide integration into the secure gdata apis.

After tagging a number of rats with the various gdata apis, we threw them in a sack, shook them up, and rat #4 exited victoriously from the rodent thunderdome.  Then we chose Google Calendar as our first integration.

Other APIs will follow, but Calendar fit our read-centric, context oriented focus for the first stab.

GData Protocol

First off, here are reference resources for the subjects that I am going to do a poor job in explaining:

• Google Data Protocol
• Atom Syndication Format (RFC 4287)
• Google Calendar API Reference
• Google Data Common Elements

If you don’t believe my lying lies, these should be considered the final arbiter of ‘what should work’—except for where I have had to compromise (I will try to explicitly identify these instances), or just pose a question to devex.

Protocol Basics

If you are familiar with RSS, you should understand the concept of feeds and entries.  Atom allows you to expand the set of elements by using the namespace feature of XML.  You may see unfamiliar elements with names like gd:who or gd$who.  These are elements from Google’s library.

OAuth

As far as the KRL engineer is concerned, credentials are handled similarly to what is used with Twitter or Amazon.

meta {
        name "google_base"
        ...normal meta block stuff that we don't need to repeat...
        key google {
            "consumer_key" : "flippty_id",
            "consumer_secret" : "S3cr3+5+uff"
            }

    }Those aren’t real values, you will have to get your own credentials from Google.

scope

Google introduces the idea of an authorization ‘scope’.  A scope just means that the OAuth token is authorized for a specific API.  As part of the OAuth dance, your ruleset needs to check to see if the ruleset has already been authorized.  Google allows the user to control who has access to the various gdata/oauth apis at their “My Account” page.  Select the “Change authorized websites” link to see (and revoke) what applications have access to their google applications.

At this point is where I remind the developer not to be a jackass.  While privacy advocates make the most noise, there are millions of people who are willing to share context (personal information) with apps that provide a useful service and eschew evil.  With OAuth, the User has the power to strip your app of any utility with a single click if you abuse your relationship of trust (ed. insert obligatory Spider Man reference here)

When you create your app, you need to specify ahead of time the what apis you want to access via oauth.

rule auth_app is active {
        select using ".*" setting()

        if (not google:authorized("calendar"))
        then
            google:authorize("calendar")
            with
                    opacity = 1 and
                    sticky = true;

        fired {
            last;
        }
    }The Twitter documentation has a detailed explanation of the OAuth process.  Here is a quick summary of what this rule does:

1. Checks to see if the user has a valid OAuth token
2. If there is no valid token, execute the ‘google:authorize’ action (fire); otherwise skip right to the next rule in the execution order.
3. If the rule fired, stop processing any more rules. When the user returns from the Google authorization page, the ruleset will be re-evaluated (see step 1).

KRL only supports Calendar right now so the syntax is fairly limited.

Calendar

Calendar requests generally contain 4 elements.  Where possible, KRL will try to provide defaults for these values, and will take care of formatting the request properly.  I tried to keep true to Google’s wording and operations, so that you would have a decent chance of looking at the Google source references and being able to reconcile with my blah, blah, blah.

1. FEED
2. VISIBILITY
3. PROJECTION
4. PARAMETERS

FEED

Calendar has 4 feeds to choose from: calendar, event, comment, settings.

The calendar feed can identify calendars the user has created and to which he subscribes.  These feeds are the ‘metafeed’, ‘allcalendars’, and ‘owncalendars’. Metafeed and allcalendars have similar information, but allcalendars also provides user preference details like calendar color, whether it is visible, and whether it is selected.  FYI: Allcalendars is also a read/write feed, but KRL does not provide write access to Google feeds.

Event returns data about individual events listed in the user’s calendars.  By default, the user is identified by the authorization token that KRL provides. It is possible to specify a user, but without a token, you can only access events with a visibility of ‘public’.  With appropriate parameters (like start-max, futureevents, and queries) you can search for specific events.  If you already know the event id, you can specify that and limit your results to just one entry.

The comment feed is available if you know the event id to which the comments are attached.

The settings feed returns the user preferences.  If you know the name of a particular preference, you can access that individual value.

VISIBILITY

Acceptable values are public and private.  Private visibility is only enabled for authorizations.  Public visibility does not require authentication, but will be unavailable if the user has disabled sharing.  ‘private’ is the default value.

PROJECTION

Projection is a clunky name for how much and what types of data you are requesting.  Allowed values are:

Projections

‘full’ is the default value for KRL requests

PARAMETERS

Parameters include the options common to all GData APIs and those specific to Calendar.

Parameters

Specifics are available from the links provided at the top of each column.  Some people may find it easier to see examples of the different queries (expressed in KRL syntax) in action

Examples

Here is an entire ruleset that grabs the events from a user’s calendar that have been updated today

ruleset a144x16 {
    meta {
        name "google_base"
        author "MEH"
        description <<
OAuth Dance
>>
        logging on        key google {
            "consumer_key" : "xxxxxxxx",
            "consumer_secret" : "xxxxxxxxxxxxx"
            }

    }
    dispatch {
        domain "nytimes.com"
        domain "google.com"
    }
    rule auth_app is active {
        select using ".*" setting()

        if (not google:authorized("calendar"))
        then
            google:authorize("calendar", {"foo" : "bbar"})
            with
                    opacity = 1 and
                    sticky = true;

        fired {
            last;
        }
    }
    rule cal_request is active {
        select using ".*" setting ()

        pre {
            today = time:now({"tz" : "America/Denver"});
            foo = google:get("calendar", {"feed" : "event", "projection" : "full", "updated-min" : today});
            title = foo.pick("$..entry[0].title.$t");
            link = foo.pick("$..entry[0].link[?(@.rel eq 'alternate')].href");
        }
                alert(link);
    }

}

Get the events scheduled for a specific timeframe

google:get(“calendar”,{“feed” : “event”,

“projection”:”free-busy”,

“start-min”:start,”start-max”:end})

Get the events scheduled for a specific timeframe and display with the West Coast timezone

google:get(“calendar”,{“feed” : “event”,

“projection”:”free-busy”, “ctz”:”America/Los Angeles”,

“start-min”:start,”start-max”:end})

Get the user’s first ten (10) upcoming calendar events

google:get(“calendar”,{“feed” : “event”,”futureevents”:”true”})

While RSS feeds can be used through the existing XML formats, the well defined nature of RSS (Channels->Items) allows for some optimization and convenience functions specific to the RSS format.

Here are some examples of the new RSS syntax:

global {
   dataset fizz_d:RSS <- "http://cyber.law.harvard.edu/rss/examples/rss2sample.xml";
   dataset feed_d:RSS <- "http://rss.news.yahoo.com/rss/mostviewed";
}

If you are not familiar with RSS, the RSS v2.0 specification can be found here.  Skipping all sorts of important stuff that you should probably already know if you are manipulating an RSS feed, feeds are composed of a single channel which contains items.  The KRL convenience functions can be grouped into feed, channel and item functions.

Feed Functions

There is only one feed operation for now.  There is only a technical difference between a Feed functions and a Channel functions.  They both take the datasource returned by the dataset as a parameter.  For the examples, I will use the variable names fizz_d and feed_d to show the mechanics, but please realize that those are just variable names from the sample.

rss:version(fizz_d)

This will return the version of RSS (0.91, 0.92, 2.0) that is being used by the feed.  The version makes no difference to the KRL functions, but is there as an indication of what content you might expect.

Channel Functions

Like the Feed function(s), channel functions take the dataset as the primary argument.

To pull out the items from a channel:

rss:items(fizz_d)

This will return an array of all of the RSS items in the feed

To pull individual items from a channel:

rss:first(fizz_d)

This will pull the first item from the channel feed

rss:last(fizz_d)

This will pull the last item from the channel feed

rss:index(fizz_d,n)

This will extract the nth item from the feed, where n is an ordinal integer

rss:random(fizz_d)

returns a random item from the feed

Selecting values from the various channel elements:

rss:channel(fizz_d,<string>)

Where <string> is the double-quoted name of the desired element; ie: “title”

Item Functions

rss:item(<obj>,<string>)

item() will accept as it’s first argument, either an RSS dataset, an array of items as returned by the channel function RSS:items(), or a single item—as returned by first(), last(), index(), or random(). Once again <string> can be any item element name.

Example Ruleset

ruleset txml {
    meta {
        name "xml test"
        description <<
             Dataset manipulation             >>
        logging on    }
    global {
        dataset feed_data:RSS <- "http://rss.news.yahoo.com/rss/mostviewed" cachable for 30 minutes;
     }
    rule txml_rule is active {
        select using ".*" setting ()

        pre {
            rss_item = rss:last(feed_data);
            title = rss:item(rss_item,"title");
         }
        every {
            notify("RSS feed: ", title)
                with
                    sticky = true and
                    opacity = 1;           
        }
    }
}