It's a common pattern in EDA applications to have a Qt-based GUI along with a Tcl shell for scripting and for access to the deeper functionality of the tool.  Often the GUI is used for viewing results and debugging, then once a reasonable design flow is established, the GUI is disabled and the tool is run in batch mode via a script.  Although as a scripting language Tcl is showing its age, its entrenched position among chip designers means it's unlikely to go away soon and we will likely continue to see this combination.  As a result it's worthwhile to consider how Tcl and Qt interact.

In interactive applications of this type you often want users to be able to switch between the terminal (which may be the shell from which they launched the tool, or a command window embedded in your application) and the GUI while using only a single thread.  Unfortunately both Qt and Tcl normally run their own "event loops" - that is to say, a typical main program looks like this:

    do_some_setup;   // user adds hooks here
    run_main_loop;   // execute commands, process events, etc. for duration of process
    return 0;

So it seems you have to pick one library or the other to "own" the main loop of your program.  Fortunately both Tcl and Qt are cooperative, and supply functions you can call to let them process any pending events.  So you can make your own event loop that calls those functions, and get the desired interactivity.  I tested this using the Qt "animated tiles" demo (in main.cpp):

#include <tcl.h>
...
// supply external event loop for Tcl to use after it initializes:
Tcl_SetMainLoop([]() {  // (C++11 lambda syntax)
                        while (true) {
                          QApplication::processEvents();
                          Tcl_DoOneEvent(TCL_DONT_WAIT);
                        }
                     });

// create a Tcl interpreter and connect it to the terminal
Tcl_Main(argc, argv, [](Tcl_Interp*)->int{return 0;});

// return app.exec();

With this approach we now have smooth interactivity between the shell from which this demo was launched and the GUI.  But wait, what's that fan sound?  Why is my lap getting warm?  Oh right...  this is a polling loop, and it never sleeps.  We are constantly checking with Qt and Tcl to see if they need to process anything.  Maybe what we can do is put a small sleep in between each iteration:

Tcl_SetMainLoop([]() {
                       Tcl_Time wakeup_period;
                       wakeup_period.sec = 0;
                       wakeup_period.usec = 100000; // 10 times per second for interactivity
                       while (true) {
                         QApplication::processEvents();
                         Tcl_WaitForEvent(&wakeup_period);
                         Tcl_DoOneEvent(TCL_DONT_WAIT);
                       }
                     });

OK, that's a bit better.  "top" is no longer showing anything particularly bad and I can't hear the fan anymore.  Unfortunately the animation no longer looks smooth - probably because it's doing all its work in 10 bursts each second, instead of spread out appropriately.

At this point I remember that underneath the hood, neither Qt nor Tcl does polling in their own event loops.  Instead, they register what they're interested in and let the operating system wake them up when something happens.  In Unix this happens through the select() system call.  What I really need to do is have Qt and Tcl work with each other so they can both go to sleep and wait for events of interest to either one of them, then dispatch appropriately.

After a look at the documentation I find that Tcl has had support for event loop integration going back to Motif.  Tcl only requires that you supply a Notifier, which is just a struct of function pointers that provide an API for Tcl to register its interest in particular file descriptors, and to register for timer callbacks.  The example I followed is here.  On the implementation side, Qt provides a nice API for monitoring activity on file descriptors through the QSocketNotifier class; anytime something of interest happens Qt emits a signal that we can listen for and pass on to Tcl.  The resulting code is here, and the main program from above now looks like:

QtTclNotify::QtTclNotifier::setup();  // registers my notifier with Tcl

// tell Tcl to run Qt as the main event loop once the interpreter is initialized
Tcl_SetMainLoop([](){QApplication::exec();});

// create a Tcl interpreter and connect it to the terminal
Tcl_Main(argc, argv, [](Tcl_Interp*)->int{return 0;});

Now the animation is smooth, the shell is responsive, and the CPU load is reasonable.  Success!

I just returned from this year's C++ Now (formerly BoostCon) conference, where one of my favorite presentations was from a couple of students at University of Potsdam, who have released an open-source numeric integration tool for ordinary differential equations, called odeint.  Their library handles any differential equation of one independent variable (say, "t") that can be written this way:

I've been meaning to experiment with Field Sets since I first read about them last year.  Because they are accessible via "clicks not code" they are more accessible to administrators, plus may be changed independent of the code in the case of a managed package. As I began to review the supplied examples, though, I became concerned - they all showed field sets being used via the '{!xx[yy]}' syntax to access the value, not the name, of the chosen fields.  I needn't have worried, though, as using this syntax directly in the text:

{!$ObjectType.Contact.FieldSets.CTFields}

resolves to a comma+space separated list of field names.  Some minor changes to my JavaScript and the controller allows it to accept either hardcoded field lists or a fieldset.  Updated code may be found on my github as usual.

For now field sets aren't available in Visualforce - just Apex - but they will be very soon.  The amazing Abhinav Gupta covers the new development as well as a temporary workaround here.

Ever since I heard about the new Force.com REST API I wanted to try it out and see if it would make implementing the data stores for the ExtJS grids I posted about previously easier, or make them faster.  I finally had the opportunity to try this out, but in the meantime the Javascript Remoting feature was released also, so I decided to implement the work I had previously done with the AJAX Toolkit using both new techniques, and then compare the results on all three.

These implementations are based on ExtJS 3.4 - 4.0 includes a significantly revamped Store concept that I'll need to investigate, but I believe the performance comparisons should be valid.

The Approaches:

1. AJAX Toolkit: my original approach.  Uses a Javascript SOAP implementation of the existing Web Services API.  It's easy to use and returns objects (SObject, DescribeResult, etc.) that resemble their equivalents in Apex, on which you operate directly.  Unfortunately it's less efficient compared to REST because of the SOAP/XML overhead.  You also have to load an extra Javascript library (connection.js) which may worsen page load times.
2. REST API: Much like the AJAX Toolkit, but uses the increasingly popular REST approach with your choice of XML or JSON for communication.  It isn't strictly REST - you must use a SOQL query to GET records - but for create, update, and delete it does a good job of matching the REST approach used in other APIs.  A bit of setup is required, though, because REST was (apparently) not originally intended for use in Visualforce pages and there is no "endpoint" on VF servers.  Instead you must use a special proxy, and enable this in your org.  Pat Patterson has a good explanation of the problem and its solution.
3. Javascript Remoting: This is a new feature which allows controller methods to be "remoted" into Javascript functions.  Primitives and return types are converted automatically for you, and it handles all the communication headaches.  You only need to supply success/failure callbacks.  In addition, this feature is acknowledged to be based on the ExtJS Direct concept, which makes it easier to integrate with ExtJS stores.  I found only two issues of usability: first, although even complex method return values are converted automatically, method arguments can only be primitives.  This means complex structures have to be rendered as strings by the caller and then unpacked in the controller.  Second, remoted methods must be static, which means there can be no per-object state.  This means, among other things, no caching of query results for paging purposes.

You can see that Remoting does quite well on the Describe call (which I use to set up properties of grid columns), but less well on the record transfer (the first grid page, of 18 rows).  Looking at the amount of data transferred is instructive:



The AJAX Toolkit and REST both deliver a fair amount of data on the Describe call compared to Javascript Remoting.  This is probably due to the fact that I only use a subset of the information available from the call.  My Describe remote method only supplies the information I actually need.  You can also see that Remoting transfers more data than REST, per page.  If you inspect the data transferred (e.g., in Firebug), you can see that JS Remoting uses a fair bit of extra formatting information, presumably to aid deserialization/error checking.

Code for the components, the JS Remoting controller, and a test page are available now on Github.

For some time I've been thinking about the Campaign tree component I described on here previously, and wondering if it would be possible to extend it, using the ExtJS Drag and Drop functionality, to provide a nice hierarchy editing feature.  Since Dreamforce '10 is coming up I spent some extra time this week exploring this question, and it turns out the answer is "yes".  The component and controller are available for your download and experimentation.  Improvements on the original include:

1. Works for any object with "Name" and "ParentId" fields.  For example, Account hierarchy can be viewed and edited instead with only a change to the component's object attribute.
2. Loads data on demand instead of at page load time.  This means the page loads faster, but each time you expand a node there will be a delay for the server to return the children of the node.
3. Uses strictly Visualforce and not the "AJAX Toolkit".  I did this as an experiment (suggested by Abhinav Gupta).  I'm still not sure which is better - this approach requires some tricky use of actionFunction "rerender" and "oncomplete" attributes that may not be as easy for readers of the code (or writers, for that matter :) to understand, but it does work, and arguably is more efficient.

Here's a simple example page using this component:

<apex:page standardController="Campaign">
    <!-- Campaign hierarchy editor using custom VF component based on ExtJS TreePanel widget -->
    <!-- by Jeff Trull 2010-12-03 -->
    <apex:form >
        <apex:pageBlock title="Campaign Hierarchy Editor" tabStyle="Campaign">
            <c:Hierarchy_Editor object="Campaign" fn="rerender_detail"/>
        </apex:pageBlock>
        <apex:actionFunction name="rerender_detail" rerender="campaigndetails">
            <apex:param name="campid_passthru" assignTo="{!campaign.Id}" value=""/>
        </apex:actionFunction>
        <!-- For some reason I have to enclose the following inside an outputPanel for -->
        <!-- conditional render to work properly.  Without it rerender still works, but does not take into account "rendered" -->
        <apex:outputPanel id="campaigndetails">
            <apex:outputText value="Double-click to display details of a Campaign" rendered="{!campaign.id == null}"/>
            <apex:detail title="Selected Campaign Details" rendered="{!campaign.id != null }" subject="{!campaign.id}" relatedList="false"/>
        </apex:outputPanel>
    </apex:form>
</apex:page>

As usual, I'd love to hear from anyone who finds this useful or has comments on the code!

I'd been thinking of making my earlier work on ExtJS grids, trees, and menus into Visualforce components and after some recent encouragement from Abhinav Gupta (whose excellent blog is a great resource for Visualforce developers) I finally got it done.  The data grid was fairly straightforward, but the Campaign hierarchy selectors proved a little bit trickier.  A way to communicate the chosen value to the user's controller or page was needed.  Finally I realized that apex:actionFunction could do what I needed - setting controller values, re-rendering sections of the user's page, and executing arbitrary Apex code, all in one.  At the same time we get a general callback mechanism.  Here's my test page for the Campaign chooser:

<apex:page standardController="Opportunity">
    <!-- A page to test using the campaign menu component -->
    <apex:form >
        <apex:pageBlock title="Selecting Campaigns via Menu Tree" tabStyle="Campaign">
            <c:Campaign_Select_Menu fn="rerender_detail"/>
        </apex:pageBlock>
        <apex:pageBlock title="Selecting Campaigns via TreePanel">
            <c:Campaign_Select_Tree fn="rerender_detail" allowinactiveselect="true"/>
        </apex:pageBlock>
        <apex:pageBlock title="Selected Campaign Details" id="campaign_pageblock" tabStyle="Campaign">
            <apex:pageBlockSection columns="4">
                <apex:outputField id="campid" value="{!opportunity.campaignid}"/><br>
            </apex:pageBlockSection>
        </apex:pageBlock>
        <!-- Create an action to update the campaign ID from the selected value and rerender the campaign information -->
        <apex:actionFunction name="rerender_detail" rerender="campaign_pageblock">
            <!-- It seems if you call this param "ID", bad things will happen -->
            <apex:param name="sekritid" assignTo="{!opportunity.campaignid}" value=""/>
        </apex:actionFunction>
    </apex:form>
</apex:page>

The apex:actionFunction tag above creates a JavaScript function called rerender_detail, which, when called, sets the opportunity CampaignId to the selected value and rerenders the campaign_page section.  You could also specify an "action" parameter which would call a controller function.

The datagrid component is used like this:

<apex:page >
    <c:ExtJS_Data_Grid_from_SObject object="Contact" fields="Id,FirstName,LastName,Birthdate,Email" rows="18" minimized="true"/>
</apex:page>

The "minimized" parameter will bring up the grid in collapsed form, which is good if it contains a lot of data that the user may not want to see (the load from the server doesn't happen until the grid is expanded).

The Campaign menu, Campaign Tree, and Data Grid components are now available online.  I'd like to hear any thoughts or feedback people have, so please comment.

I first got involved in Force.com because I was volunteering at a local nonprofit that was moving from a customized database system running on their office machines to a Salesforce.com instance based on the Nonprofit Starter Pack (NPSP).  Although I eventually became more involved in the implementation, my primary task was helping them move their data to the new system. I've now learned a fair bit about how this works, so I'm documenting my experiences here in case it helps someone else.

Data migration initially seemed like a fairly straightforward task.  As it seemed to me (and was described by others) it was merely a matter of determining, for each field in each table in the existing system, which tables and fields in the target system to transfer the data to.  Any problems that cropped up in this process could be "cleaned up" by sophisticated tools once the data was present in the target system.  As such, it was conceived of as a largely mechanical process.

Here are some of the ways in which this model broke down:

1. The source and target system schemas were different in important ways.  Sometimes a 1:N relationship in the source became a 1:1 in the target, and some information was simply dropped in the process.  In other cases an data value (flag, picklist value, etc.) in the source became a relationship (e.g., campaign member) in the target.  Sometimes a variable length list in the source (e.g. email addresses for a contact) became specific field values (home, work, other email) in the target. Sometimes a source object could become either one or another kind of object in the target system, depending on the presence of, or values of, other fields.  Finally, there was some additional data to supply - we turned a flat data structure in the source system into a tree (of Campaigns!) in the target with an additional input file.
2. Sometimes complex heuristics (supplied by the organization) were required to determine how to classify records.  For example, depending on the circumstances, a contribution from a Contact could be regarded as a donation or as payment for their membership.  This is a very important distinction for the users.
3. Often the data in the source system is found to be bad in some way, and needs to be flagged in a helpful manner for cleanup.  Other times a heuristic can be applied to fix the problem in flight.
4. Sometimes the target system's business logic (esp triggers and workflows) are counterproductive.  For example, in the NPSP as we have it configured, creating a Contact also creates an Account and a Household object.  Careful sequencing is required to ensure that Contacts that need to share a single Account or Household do so.

Finally, a manual process would have caused considerable pain because the upload process, requiring human intervention, would have been long and error-prone.  Instead, I ended up writing a Perl script of about 3500 lines to automate the migration process.  Here's how the general flow looked:



The basic steps are:

1. Dumping data out of the source system, which in our case was FileMaker Pro.  I found that creating custom table layouts that reflect the values you intend to export is very helpful, because FileMaker will volunteer to use the displayed columns automatically during XML export.  Excel and CSV export is also a possibility, but we had too many records for Excel (64,000 limit) and I ran into issues with quoting when using CSV.  At the end of this step we have one XML file for each table we export. (This step is partly manual).
2. We then parse these XML files one at a time, beginning with the tables that will become Accounts and Households, using the XML::LibXML package from CPAN.  The result of the parsing process is an array of hashes, each representing a line in the input table.  Semantic errors peculiar to each table are checked for and either fixed with heuristics or just reported and skipped, and new data structures are built containing the intended target data.
3. We then begin a cycle of uploading tables to the target system using a combination of the Bulk API and Web Services API.  For Perl users there are far better examples of the latter online, but our bandwidth needs mandated I investigate and implement a Bulk API approach.  After each upload completes, we acquire and store the newly-created IDs for every uploaded object so they can be supplied in later uploads to indicate references from object to object.  Typically in those cases there is some key already present in the source system that I can leverage to create this mapping.  The next table to be uploaded receives these new values, and the cycle continues.

For those thinking of using the Bulk Data API I have some tips:

• Carefully consider how many records you put in each batch.  There is a maximum of 10,000, but you can run into problems with far fewer if each record takes a long time to process.  Batches that take over 10 minutes to run may sometimes complete, but they tend to be unreliable and in my case, triggered some Salesforce back-end bugs of the "unexpected error, contact support" variety.  Batches that are too small, however, may cause you to exceed the batch limit (500 per day on our instance) or the API request limit (10000).  I experimented until I got a runtime of about 6-7 minutes per batch on each object.
• Use parallel batches if you can, as it can make a huge difference to runtime.  This worked for some of our objects, but on others the batches would time out with failures due to lock conflicts.  This seemed to be due to triggers and rollup fields in other objects referenced by the one we were uploading. For some of the objects that failed I could work around the problem by sorting the records using, as a key, the ID field that references the object with contention.  This tends to keep potentially conflicting records together in the same batch.
• Use compression on your HTTP requests.  I didn't do this at first (thought maybe the library handled it for me) but it was very helpful for larger objects.  XML is highly compressible...

I've put a copy of my Bulk API upload code here; hopefully someone can benefit from my experience.

In my last post I presented an approach to supplying data to an ExtJS data grid that uses the Force.com AJAX Toolkit to supply data to the underlying store in an on-demand fashion, instead of once when the page is loaded.  This approach has some advantages over the static <apex:repeat> technique, but is still deficient in a couple of areas:

1. It does not provide for modifying the data.
2. It is not well integrated into Ext - updating data from the server will require some complex programming to recognize when the store should be reloaded.

After some thought and a review of the documentation I concluded that the right approach was to model an Ext "Store" with code underneath to perform the CRUD actions via AJAX Toolkit calls.  With the hooks provided by Ext and the AJAX Toolkit, I could even make everything work properly in an asynchronous manner with components refreshed as data arrives, events propagated to the right points, and so on.  In addition, with Force.com objects now living as first-class citizens in the Ext ecosystem, lots of cool new integration possibilities would be opened.  I had done something similar before for another system, so it seemed quite doable.

I began coding with the plan of implementing a DataProxy subclass, but after nearly completing it I discovered Ext Direct, along with the related classes DirectProxy and RemoteProvider - new features in ExtJS that allow server methods to be exported easily to clients.  Although the AJAX Tookit isn't perfectly compatible with Direct, there is a good fit on a structural level, and I found that I could subclass DirectProxy and make it work with a small amount of changes.  I also needed to ensure that the store using the new proxy is configured a particular way, so now the high-level interface is a new store I'm calling SFDCStore.  The result is here.  To demonstrate the capabilities of this Store class I've added the following features to the grid demo:

• Insert, edit, and delete operations
• Server-side sorting and paging

Comments and feedback are appreciated.

I've been trying to integrate the amazing ExtJS library into some Visualforce work I've been doing, to create slicker-looking interfaces that better mimic the system a client was migrating from.  Like others, I discovered the Appirio blog entry describing a technique for taking controller data and using it as a data source for an Ext GridPanel (or indeed, any Ext component that can work from a "Store").  Basically this technique works by unrolling, with apex:repeat, a list of records from a Salesforce object and initializing a Store with that data.  It works pretty well for data that changes infrequently, or that you are willing to reload the page to update.  For example, the Campaign hierarchy chooser I built relies on this mechanism to initialize the tree.

Unfortunately this approach has some limitations:

1. The data is static: it does not change unless the entire page is reloaded.  This means we cannot make changes to the data behind the scenes and then rerender only the associated grid, for example.
2. The data is monolithic: it must be loaded all at once.  This may slow down page loads, or in the other extreme, mean that less information than desired is displayed.  Furthermore, the data is loaded even if will never be seen by the user (it is hidden or off-screen).  All data that may be needed until the next reload must be generated and transmitted by the server.  Paging and sorting must happen on the client side.
3. There is no mechanism for letting the user make changes to the data.  This method is strictly read-only.

I initially had hoped to use Visualforce components and some controller code to get around the first two issues.  With apex:actionFunction and apex:param you can wrap JavaScript functions around controller methods.  Unfortunately I found it difficult to cleanly implement what I wanted this way.  Perhaps someone will post a good solution.

I found the cleanest (and hopefully most efficient) way to do what I wanted was to make use of the AJAX Toolkit provided by Salesforce.com.  This is a JavaScript wrapper around the Web Services API that allows you to do much of what you can do in Apex directly in JavaScript.  Using this approach I was able to load data after the page loads, in response to user input.  My modified version of the Appirio example is here.  The same grid is produced, but the "collapsing" functionality provided by the grid is used to postpone data loading until the user expands the grid using the small button in the upper right.

As for the third item noted above, I have an idea in mind which will hopefully be the subject of a future post.

This was a relatively short day for me, as I was trying to consolidate what I'd learned, and work a bit on my Spirit LEF/DEF parser.  Here are the sessions I attended:

• Joel Falcou and Jean-Thierry Lapreste: "The Numeric Template Toolbox..." A library for doing various types of matrix operations, modeled after (and using nearly identical syntax to) Matlab.  They also did a fair amount of investigation of incorporating SIMD instruction sets like SSE and Altivec for performance.  This may end up being a separate library.  This talk was very well received.
• Eric Niebler: "Practical DSEL Design with Proto". Proto is one of the libraries Spirit is based on.  Eric took us through some simple (std::map initialization a la Boost.Assign), and then far more complex (Boost.Fusion substitute) designs using his library.  You can use Proto to create a checked AST from users' inline C++ code using overloaded operators, rearrange that AST, and have the compiler generate suitable code.

My brain is just about full now, but there's only a handful of sessions left, on Friday morning.