Large Data Sets in Salesforce Apex (a.k.a Why the fuck is this query timing out?)

Topics included included in this post:

• Javascript Remoting (if your end goal is Visualforce)
• Test Data
• Analyzing Performance Gains
• Optimizing Your Where Statement
• Use Aggregate Queries
• Streamline Data Returned
• Use Recursive Logic to Break Up Unwieldy Queries

This post is for anyone struggling to work with large data sets in Apex, particularly if you're aggregating that information in some way, but some of this can be useful even for returning detailed records. I've spent the last few months creating a Visualforce reporting page that aggregates just about every record created in a selected time period in my client's Salesforce instance.  When I began the project, they were having trouble pulling data for their larger offices even one office at a time, and my goal was to create a page with tabs for various metric categories (revenue, new/lost customers, payroll, survey feedback, lead and marketing spend analysis) and in each tab, pull all offices' (a little more than 200) metrics together for comparison.

Their business deals in a high volume of relatively small and often recurring transactions with customers.  To give you a rough idea of scale, a single month's worth of revenue data is spread across about 70,000 records, and they were hoping to run a report for a 6 month time span.  Some of the metrics they wanted required analysis deeper than simple sums and averages, notably customer attrition. All of this was further complicated by the fact that they need to group the data by office, and many of those offices are comprised of multiple locations whose metrics need to be rolled up to a parent office.

On to the meat of it.  Few of the tips below for optimization are new, many are best practices or tips that other Salesforce experts have already put out there, but I wanted to compile them together in one place with some additional practical advice on how and when to use them.

Javascript Remoting (if your end goal is Visualforce)

Probably the most obvious piece of advice on this topic.  I won't go into the technical details too much here since there's a wealth of information already out there, but the advantages come from the asynchronous nature of the remote calls to Salesforce (more tips on that below) that mean you can break up your calls to Salesforce into manageable chunks and the fact that you don't have to deal with the Visualforce View State (hooray!).  As awesome as that all sounds, it also means you'll need to be fairly adept at Javascript and you're on your own for generating the UI, no Visualforce tags to automatically render and format everything prettily for you.   

A few good resources:

• Salesforce Visualforce Remoting Documentation (you'll need to either "Next" your way through it or navigate through the menu)
• How To: Visualforce Javascript Remoting by Craig Isakson: a high-level overview of the hows and whys.
• Javascript Remoting in Salesforce: A Primer by Derek Lansing: the first of a 4 part series that gets more in-depth.

Test Data

Ideally, you should be building this in a full sandbox. If your requirements are such that you need advice like this to achieve your goals, you have a sound argument for your employer / client as to why they should spend the money paying for one.  If that's not an option, you'll at least want a Developer Pro sandbox with as much test data as possible, but be warned that your page may work beautifully in the sandbox and then crap out when pulling real data in production.

Analyzing Performance Gains

Let me caution that some of these tips may work in some scenarios but not others.  Removing a non-selective filter from a query and applying that filter logic to the results returned instead may help with one query, but hurt the performance of another query (that was my experience).  So you'll need to get into your debug logs and look at timestamps to verify which method of getting to the results you need works best.  And if javascript is doing a lot of the legwork, you might need to sit and time things with a stop watch (I did).

And on that note, be aware of Salesforce's Query Cache.  The first time you run a query, Salesforce is putting the results into a cache that will stick around for a couple hours.  So the next time you run the same query, even if you've made no changes, you'll find that it runs lightning quick.  This means you'll have to keep adjusting your query criteria (the easiest option being to query a different date range) to ensure any performance gain you're seeing is due to optimizations you're making and not that cache.  There were times I thought I made grand successes, only to have the same query time-out the next day when the cache had been cleared- disheartening.

Optimizing Your Where Statement

I was already familiar with many of these concepts, but there were queries with time-out issues that were extraordinarily hard to resolve that taught me a few new tricks.

A great resource on this is the Dreamforce presentation "Faster SoQL? Yes, Please" by John Tan (@johntansfdc) and Chris Peterson (@ca_peterson), which includes some of these tips and more.  I wish I had happened upon this before I learned some of these the hard way! At the risk of flooding them with requests for help, those two are the true query optimization gurus.

Making Filters Selective

A selective filter is one that leverages indexed fields in Salesforce, which can significantly improve performance.  The "Make SOQL query selective" page details out what this means, and also points to some additional resources. If you're having issues with a query that is filtering by a field that's not indexed by default (like Lookups and External Id fields are), you may be able to request that Salesforce add a custom index to that field (this is restricted to certain field types).  Use the Query Plan tool in the Developer Console to help tune your queries and identify fields that may require custom indexing.

I also just discovered another tool available if you want to dive in even deeper- the Query Resource Feedback parameter available in via the REST API.  This will actually detail out how the Salesforce query optimizer is evaluating each criteria in your query.  An overview and links to additional resources can be found here on the Salesforce Developers site.

Remove Non-Selective Filters- Non-Aggregate Queries

The flip-side of the advice above- if all of the filters in a query are selective except one or two it may be more effective to take those non-selective filters out and manually filter that data out of the results. That's a little abstract, let's try an example.  I had a few queries with multiple criteria on indexed fields that also included a filter on a picklist field (where Status__c = 'Active').  I altered that query to remove that Status criteria and instead group the results by Status.  I then looped through the results and compiled the data just for the results where the Status was "Active".  The end results were the same, but time to get to that result dropped by about 30 seconds.  This same advice can be applied to any not (<>) criteria you may have, which isn't selective even if it's on an indexed field.

Be aware that what you're hoping for, but may not necessarily get, is a trade off in the reduction of time to run a query that makes the additional time manually processing the results worth it.  It might be that your data returns so many results that any performance gain on the query itself is completely lost as your code loops through and processes those results.

Remove Non-Selective Filters- Aggregate Queries

Same idea as above, but in aggregate queries you have the option of migrating those non-selective filters from your Where statement to a separate Having statement. More details here.

Add Selective Filters, Just Cause

You might be able to boost the performance of a query by adding a selective filter that logically speaking you don't really need.  The simplest example of this would be to add a filter on a RecordTypeId field, even though your other filters could only possibly return that Record Type anyway.  A more elaborate example.  I worked with a custom object used to store details of payroll that included a master-detail field to the Contact object.  The only Contacts that can possibly be in that field are employees, so I didn't need to include that as a criteria in addition to the date range.  But, running a separate query to generate a Set of employee Contact Ids and then including that as a filter in the Payroll query notably improved performance.  Go figure.

Nix Those "Or" Statements

"Or" statements that include fields can kill query performance, so you want to get rid of them wherever you can.  This is easier said than done; sometimes life demands complex boolean logic.  In the scenarios where I had no choice but to include some "or" logic, I ended up breaking the query up into separate queries and then compiling the results together.

Use Aggregate Queries

This may also be obvious depending on your database background, but whenever possible let Salesforce's query engine do the heavy lifting on grouping and mathematical operations on your data.    For some of the metrics I was generating, particularly related to customer attrition, it was impossible to get the results with a single aggregate query.  Even so, a series of aggregate queries used to gather the criteria needed for the final metric got me there.  And for metrics where I was able to shift from the old method of looping through individual detail records and compiling data for calculations, to an aggregate query that summed/averaged it all up for me, the performance gain was extraordinary. 

Aggregate SoQL Resources:

• Salesforce's SoQL Group By documentation paired with the Aggregate Function documentation
• Useful SoQL by Chizuru Yamamoto has some useful examples.

Field History Conundrum

A very specific issue, but one I suspect other people have struggled with as well.  There were a number of metrics which required determining a customer's Account "Status" (custom field) or Job "Status" at a specific point in time.  So their Status may be "Canceled" right now, but I needed to determine if they were "Active" 6 months ago.  That information was available in Salesforce's Field History tracking data, but was rendered effectively useless by two limitations of the Field History data:

• You cannot do Aggregate SoQL on Field History data
• You cannot filter Field History data by the New Value that's being tracked, only by Field Name.

So to get the very basic information of which customer Accounts were active on a certain date, their old logic had to loop through hundreds of thousands of Field History records to pick out the ones that were active on a certain date.  It barely worked to pull information for just one office, and was completely unfeasible for pulling all offices at once.

So I, annoyingly, recreated that field history logic with a new trigger.  It works almost identically to the standard field history, but I limited it to only track the Status field and added in a lookup back to the previous Status History record so there was a quick way to find customers who, for example, went from Canceled status to Active status.  The trigger was pretty simple, but the downfall was a ton of new data in Salesforce- the built-in field history tracking doesn't count toward data storage space but the new custom object obviously does.  For the new reporting capabilities it provided, they decided it was worth it.

Streamline Data Returned

You can lose performance on the browser side if you're returning more data from a @RemoteAction call than you really need.  If you're returning SObject data, make sure it doesn't include extraneous fields and same if you're returning a custom Apex class (which is converted to a JSON object in Javascript).  While you're testing, it's worth throwing the object returned into a console.log statement in Javascript so you can do a sanity check that it's as efficient as it can be.

Use Recursive Logic to Break Up Unwieldy Queries

This should be a last resort after you've done every optimization you can come up with, but let's say you have and you're still hitting time-out issues in Visualforce (I've had and solved similar issues in triggers and Batch Apex like this as well).  For each of the blocks of metrics I had to return (revenue, payroll, etc), I had to painstakingly figure out the maximum date range that I could return at once. And when determining that, don't forget my note above on the Query Cache. I then defined the logic to break up the date range selected by the user into manageable chunks.  

So at the end of one RemoteAction call, the logic checks to see if additional calls are required and if so, recursively jumps back and makes the RemoteAction call for a new date range- all the while merging the results into one persistent Javascript object.  A more concrete example.  The user selects a date range of 6 months to return revenue metrics, which I determined can only be returned no more than 31 days at a time.  The Javascript makes the same call 6 times, once for each month.  I actually have a quick bit of logic that always restricts each call to occur in the same month, since that's how the results are reported so it's just easier than grouping the results by month in Apex.  Once everything has been queried and compiled, the logic to actually render the results on the page is called.  

In some scenarios, I was able to break the logic up into multiple calls- one for marketing spend, one for new leads, one for new recurring customers- and then compile and analyze that data with Javascript.

This method is far from ideal and bear in mind that performance becomes very dependent on the user's machine.  So it may work fine on your slick development computer, but if any of your users aren't as technologically endowed, they're going to sit and wait longer while their browser processes the results.  But if you've really explored all of the options of optimizing the Apex and are still hitting time-outs, this could be your saving grace.  

I'd recommend setting this up so you can easily change the amount of data called at once (date range is the easiest option if you can) since you may need to adjust it as either the data expands or (hopefully) Salesforce performance increases.  I have one variable where I set the number of days to be called at once, and a quick routine that sets that variable based on the type of metric being returned.  

Amazon.com knew I was pregnant before my mother

All it took was a purchase of one nursing bra and Amazon began flooding me with advertisements for pregnancy and baby-related products.  Some people find this kind of targeted advertising convenient.  I find it invasive and annoying.  I don't even mind it if I'm outright searching for something and they show me related products, but now whatever I'm doing in Amazon, I'm bombarded with suggestions on what I might want for my breasts, nether region or yet-to-be-born baby.

With the "convenience" that comes with having your every digital move catalogued and analyzed, we lose a significant amount of privacy.  Today it's my own view of Amazon, Google, Facebook, etc. that's affected by my purchases, searches and comments; what of the day when Amazon starts recommending to my friends and family viewing my Wish List that I might appreciate some nursing pads or stretch mark cream?  And if you think they won't, you're taking the concept of "corporations are people too, my friend" too seriously.  I'm sure there are plenty of Amazon employees and even executives who would oppose such tactics, but ultimately it's the bottom-line that will drive such decisions; after all it's their legal obligation to their shareholders to scrabble for every dime they can.

I quit Facebook the day I commented that I was going to Boston and started getting advertisements for Boston restaurants and events in my sidebar.  I knew that day was coming though and had always been conscientious of what I posted there.  I don't think people, particularly young people, are always that aware.  Once it's online, assume it's there forever.  Go ahead and delete or close your account thinking that's wiped it out, it's still on some server somewhere.

Think Facebook won't turn over a log of your full history (deleted or not) to a government intelligence agency if asked, even without a warrant?  Did you think the same thing about phone companies before the Bush administration made it clear that saying the word "terrorist" absolves both the government and telecommunications companies of having to adhere to any laws, right down to the very Constitution they supposedly hold so dear?  And it's not just the government you have to worry about, Facebook has already "accidentally" revealed what was supposed to be private information of users to third-party application developers when those users installed their apps.

Whatever privacy scandals we hear about related to social media and search engines (try googling "Google Germany Privacy" for some insight on how they push the boundaries) is only what they've been caught doing and assuredly just the tip of the iceburg, and advances in technology and the evolution of artificial intelligence will bring about privacy breaches we can't even yet fathom.  It's a strange, new world we live in.

There is a bit of "turnaround is fair play" going on in the world though.  The same companies that push the boundaries of the trust we put in them with our personal data have put their own trust in the government and companies of China as they use factories there for manufacturing and setup branch offices.  China doesn't have the same ideas about intellectual property as we do.  Companies have found themselves victim to corporate espionage, stealing and recreation of their designs and even disturbingly finding monitoring devices built into electronics that were not in the design.  The Chinese government offers lip services to such breaches, but rarely takes any action on the matter.  Be patriotic and come back to the USA guys; we may expect living wages and safe, non-toxic work environments, but at least you can have us arrested and sue us if we steal from or sabotage you.

Zoo Life- The End

My stint as a docent at the Prospect Park Zoo came to an end a few months back and I thought I'd take a moment to talk about why.  Partly just for the irony of it; you'll see what I mean.

Here's how the story began, but not why I ultimately quit.  I was part of the animal handling program there.  This had been going on for a number of years; docents who had gone through training and been certified were able to present animals to the public during tours and scheduled presentations on  weekends.  Not all of the zoo animals mind you, only ones the animal keepers had designated as tractable based on species and the individual, like chinchillas, chickens and some snake and lizard species.  It was a rewarding program for the docents and a nice chance for the public to get up-close-and-personal with some of the animals, including a chance to touch them.

Rather suddenly, the program was ended.  To my knowledge, there was no unfortunate incident that brought this on, and that's the kind of rumor that spreads pretty quickly.  This seemed more like a policy decision from the managerial level, probably from someone new to the zoo who was worried about liability.  That's a guess though, we were just told that the program was no more.  I was disappointed, but also recognized that I was just a volunteer there and as such didn't have any say in policy.  Some of the other docents took it a lot harder, there were meetings (including tearful pleas) and emails and petitions on the web to appeal the end of the program.  Frankly, it was all a little melodramatic for my tastes.

At this point though, I'm still a loyal volunteer and staying out of the fray.  I even received a call from the NY Post (blech) on the matter and had nothing but good things to say about the zoo.  Until the day I was informed that a new condition of being a volunteer was that I had to sign an agreement saying that I would never make any public (via media, Facebook, email or however) negative statements about the zoo.  Excuse me?  I had no problem with restrictions about what I could say while I was at the zoo acting as a representative, but trying to restrict my free speech as an individual outside of the zoo?  That was unacceptable to me.  If I was an employee, it would be illegal to require that of me; we have whistleblower protection laws in this country for that.  If they did ask their employees to sign the same agreement, I hope the illegality of it comes back to bite them in the ass later.

I don't like it when people try to inappropriately assert authority over me, or anyone.  People who do so are generally bad human beings even beyond their pathetic little power grabs.  Being the CEO of my own company has made me rather sensitive to it.  So told them I wouldn't sign and why and left the zoo on my final day of volunteering sniffling as I did so.  I really did enjoy my time there until some asshole  who didn't even have the courage to identify themselves came along and ruined it for me.  God I hate having principles sometimes.

The Joy of Salesforce Governor Limits

Working within the governor limits in Salesforce can be a real balancing act.  I understand why they're there; it's a shared environment and you can't have developers writing code that bogs down the servers for everyone else. Especially because, let's face it, there are lot of really shitty developers out there. But that doesn't mean I don't curse their name when I see those dreaded LimitException errors.

I started writing Apex as soon as it came out; back when the governor limits were a lot more stringent and nifty tricks for avoiding them like future methods didn't exist yet.  Nowadays, I have a number of clients processing large amounts of data and/or implementing very complex business processes on their data, so I've come up with quite a few tips and tricks on dealing with the limits that I thought I'd share.

Bulk Proofing Code and Future Methods

I'm lumping these together because you don't really need me to explain them to you.  Unlike many other bloggers, I'm not interested in proving how clever I am by regurgitating information already published much more eloquently by Salesforce themselves.  So instead I'll just refer you to them-

• Future annotation
• Bulk Trigger Idioms

If you want more info on those, Google is your friend.  If you can't figure out how to bulk-proof your code, you probably have no business writing code; this is shared environment coding 101 (and is not so bad for developers working within their own environments to know either).  

Two drawbacks of Future methods that are worth noting if you're considering implementing them-

• Your future method may be generating records or updating fields that your users are expecting to see instantly, and it can be confusing for them if they don't. Generally it happens fast enough that all they have to do is refresh the page to see it, but asynchronous logic is a little abstract to explain to a non-developer (and even to a lot of developers).
• If your future method hits an error, either your user will never know it or you'll have to write in logic that emails them if an error is hit- but again, kind of confusing to an end-user.  Be prepared to have to fix errors being hit in future methods that you receive by email and remember to go back and clean-up any data affected by the fact that your method didn't fire correctly.

Chunking Future Methods

I've had scenarios where I have DML operations on multiple records, either set off by Apex or data imports, that fire a future method and then hit governor limits within that method.  Each time you run code, you can set off up to 10 Future methods so if governor limits are an issue in your future method, consider breaking up whatever you're sending into the method into manageable chunks.  An example of some trigger logic that does exactly that here-

//Breaking this into chunks of 30 to avoid governor limits

Map<Integer, Set<Id>> mapSetStats = new Map<Integer, Set<Id>>();

for (Attendance__c a : Trigger.new){ 

    if ((a.Status__c == 'Attended' &&Trigger.isInsert)

     || (a.Status__c != Trigger.oldMap.get(a.Id).Status__c 

         &&Trigger.oldMap.get(a.Id).Status__c == 'Attended')

    ){

    //for first attended or when last Set in the Map has 30 ids, add a new Set to the map

    if (mapSetStats.size() == 0 || mapSetStats.get(mapSetStats.size()).size() == 30){

    mapSetStats.put(mapSetStats.size() + 1, new Set<Id>());

    }

    mapSetStats.get(mapSetStats.size()).add(a.Id);

    }

}

//Sync Monthly/Annual Client Statistics:

if (mapSetStats.size() > 0){

for (Integer i : mapSetStats.keySet()) 

            clsAttendance.syncClientStatistics(mapSetStats.get(i), new Set<Id>());

}

In this example, "mapSetStats" is populated by individual Sets of Ids, each of which is no larger than 30 records each.  I happen to know that users will never update more than 300 records at a time (10 Future calls * 30 record batches) so I can get away with this.  If chunking isn't feasible for any reason, then you'll need another solution, such as....

Business Logic in Scheduled Code

I've used this not only for getting around governor limits but also for limits on the other side of external web service callouts. Here's the basic idea-

• Instead of sending records to a Future method for processing, find a way to gather records that require processing in a query.  In some scenarios, I've created a checkbox field specifically for this purpose and updated that field in a trigger to true, but sometimes the very nature of what you need to accomplish (e.g. a blank field that needs to be filled in) means that you don't have to explicitly update it to identify those records in a query.
• Create a Batchable class which queries for the records that have been marked in the step above.  For details on this, see Salesforce's article on Using Batch Apex.  
• Create a Schedulable class which performs your logic on records from your Batchable class.  Again, you don't need me to figure this out, see Apex Scheduler.
• When you use Database.executeBatch to get records from the Batchable class, you can pass in an optional second parameter to set the limit on the number of records returned- you just need to figure out what's the largest number of records you can process at once without hitting the limits.
• If you did need to explicitly mark records for processing, like my checkbox example above, make sure you un-mark them when the processing is complete.  Otherwise, the same records will just keep being update in an endless loop.

As with Future methods, users will need to understand that there will be a delay in the updates that happen as a result of scheduled code.  I've been lucky enough that in the cases where I have had to use scheduled code, users haven't had an issue with that.  If your users do have a problem with it, remind them that the fancy, magical updates they're looking for were probably absolutely impossible for them to accomplish just a few years ago so they'll have to shut up and deal.  Depending on your relationship with them, you might need to re-phrase that.

Using Third-Party Generated Certificates in HTTPRequest Calls

I've been doing various integrations with external web services lately and have learned some things along the way. The latest integration I did required two-way authentication.  The Salesforce documentation makes some assumptions about these integrations-

1. That the service you're integrating with provides a parseable WSDL.
2. That the service you're integrating with allows you to upload a signed certificate that was generated from Salesforce for the two-way integration.

And if both of those hold true, the Salesforce documentation (and various related blog posts) will be sufficient for you.  However-

1. Not all services provide a WSDL, and even those that do might include XML elements that are not supported by Salesforce.  In this case, you're on your own for writing an Apex class to interact with the web service.
2. Some services provide a WSDL that Salesforce can parse, but you'll find that you need to edit the resulting Apex class to include properties that were excluded.  Though I didn't dig in to see if this was the fault of the WSDL file itself or the parser, I suspect the latter.  How to go about editing that generated Apex class is worth it's own post.
3. Some services provide you with a client certificate file that you need to include in your request- you don't have the option of generating one in Salesforce and getting it signed.

None of these are insurmountable, but they'll take a lot more leg-work on your part and in the case of having to provide a third-party generated client certificate in an HTTP request, I found myself having to piece together how to do so from various forum posts.  So I'll focus on that today.  

So an external service (in my case, the credit card processor First Data) provides you with a client certificate file and a password for that file.  Now what?  In Salesforce's documentation for the HTTPRequest class you'll notice a method called "setClientCertificate" with a note that this method is deprecated and you should really use "setClientCertificateName".  But if you've been provided with the client certificate, "setClientCertificate" is what you'll need.  Here's how to get it to work-

• Upload the client certificate as a Static Resource
• In Apex, query for that file and base64 encode the body of the file into a string variable.
• In your HTTPRequest variable, use the setClientCertificate method with the base64 encoded certificate as the first argument, and the certificate password as the second argument.

Oh sure, it's easy when you know how to do it.  And a note that alternatively, you can base64 encode the certificate yourself and use the resulting string in setClientCertificate, but the above seemed a little more elegant.  The actual code to accomplish this-

String cert;

for (StaticResource sr : [Select Id, Body from StaticResource where Name =: <certfilename>]){

cert = EncodingUtil.base64Encode(sr.Body);

}

HttpRequest hReq = new HttpRequest();

hReq.setEndpoint(<url>);

hReq.setMethod('POST');

hReq.setClientCertificate(cert, <certpassword>);

If you were lucky enough to have a WSDL that Salesforce was able to parse, you'll be setting the "clientCert_x" variable of the stub to the base64 encoded string and the "clientCertPasswd_x" variable to the password.

Once again, hope this saves someone some time!

Encrypted Fields in Apex- one gotcha

I've been doing some work integrating Salesforce with a credit card processor and hit upon a issue with encrypted fields in Apex that I haven't found documented anywhere.  It may actually be entirely intentional, but without documentation it's confusing and I did see a couple other people hitting the same problem.  According to Salesforce documentation, when you work with an encrypted field in Apex, the code will always see the unmasked value of the field.  But there's an exception, if you pass an Sobject into a method, you'll find that you'll be retrieving the masked value of the field.   I've only tested with static methods so far, and it might be specific to those.

More concrete examples.  You have an Opportunity that has already been saved with a value in the Credit Card Number field, which is encrypted.  You pass that Opportunity record into the following method.

public static void EncryptedExample(Opportunity opp){

String strCC = opp.Credit_Card_Number__c;

System.debug('Can the code see the masked number 9 in this field? '     
                     +strCC.contains('9'));

//No, it can't!

}

This is only an issue if the SObject you're passing in has been saved already.  If you pass in an SObject that either hasn't been inserted or if the value of the encrypted field has been updated but not committed to the database, then that field hasn't been encrypted and masked yet and the code therefore sees the actual value.

The solution?  You'll have to query for the record within your method and then all works as expected-

public static void EncryptedExample(Id idOpp){

Opportunity opp = [Select Amount, Payment_Type__c, Deposit__c, Credit_Card_Number__c, Credit_Card_Exp__c from Opportunity where Id =: idOpp];

String strCC = opp.Credit_Card_Number__c;

System.debug('Can the code see the masked number 9 in this field? ' 
                     +strCC.contains('9'));

//Yes it can!

}

Hope this helps someone, I know I spent a couple hours baffled (at first I thought it was a mistake in my webservice callout).

Happy Terrace Garden Time

In looking back on my last two posts, they're a bit preachy and serious; time for something lighter.  This year's garden.  Last year was our first attempt at a garden on our terrace.  This year we had learned our lessons on what plants can take our windy environment and how to best care for the plants we have.  I won't blather on- have I mentioned that when I write an instant messaging widget I'm going to call it Blather? The pictures are more interesting but a quick overview of our crops-

• We upscaled last years upside-down tomatoes in a coconut fiber basket to a bigger basket (16"), and eventually a coconut fiber cover that helps keep the moisture from being blown away.  The roots from the tomatoes grow up into the fiber, it's interesting to peak under.  
• The garlic chives from last year (in the smaller basket where the tomatoes used to be) survived inside for the winter and share the basket with chocolate peppermint, which can be a nice touch in my coffee some mornings.
• There's a golden raspberry bush, which I had never heard of before, but they grow stouter than their red raspberry cousins from my childhood that would be a bit of a monster on our terrace full grown.  
• I bought a kumquat tree last fall and we had it inside for the winter.  It had kumquats when I purchased it, which we consumed over the winter, so this is the first time I get to see them growing from flowers.
• Carnival blend carrots, which will grow as purple, red, white, yellow and orange.
• Hot cherry peppers, which have turned out to be very tolerant to the wind.
• Strawberries in an upside-down commercial bag, hung inside of a reclaimed old-school metal aquarium platform.
• In the earlier pictures is a venus flytrap, which provided garden justice to a couple of plant-eating pests, but sadly now needs to be replaced.  

Late May, the Early Days

Overview of almost everything

You can see one kumquat here, it held on since last fall when I bought the tree

Top-down view into the raspberry water bottle / greenhouse. Wasn't it cute?

Mid-June, the flowering begins (as do excited visits from bees)

And if you look just above the basket in the center, the Empire State Building.  Much better in person

Once the raspberry outgrew it, the hot cherry pepper plant spent some time in the greenhouse

The first of the tomato flowers

Rokku enjoys the garden as well

 Over time, we gave all of the plants watering spikes with their own supply of water, it got them through the hot, windy days of July while still yielding us plenty of fruits and veggies to eat.

Miscellaneous

For awhile, the venus flytrap was magical at night.  The hope was to attract bugs.

In the day, our sentry bee stood guard

 The fruiting

Isn't it beautiful?

Rokku always gets first dibs

Except on the hot peppers

Yay!

The kumquat begins to flower

The mint expands outwards

August

We've had a few tomato, garlic chive and hot cherry pepper meals from our garden at this point.

The golden raspberry has a decent number of flowers now

Little baby kumquats!

And Rokku heads off into the sunset

If you say "social media" just one more time!

As a tangentially related aside to the following rant- I'm really sick of the high percentage of "experts" I see on TV brought on to talk to me about technology, though they have zero background actually working in technology themselves.  And I'm talking about working in a capacity that creates technology, not just that they've used technology to blog or tweet before, or that they had an idea that they then hired actual technical people to implement for them.

It's too much with the social media.  Social media is just one small part of a larger culture of technology, and frankly not as earth-shaking as our "experts" would have us believe.  And if you don't believe that, go check the price of Facebook's stock.  It hardly even matters when you read this, that will likely hold true. And this is a company that gets non-stop, 24-hour, free advertising on television, radio, web sites, business cards... I could go on for awhile here.  But still you can't gain any traction in the market?

The only thing that surprised me about the Facebook stock plummet was how quickly it happened. I figured the bubble would hold for at least a couple months after the IPO. But despite the hype around the launch, corporations, and in particular publicly-owned corporations, are about profit and the Instagram purchase was just one of many cracks in the facade of Facebook's potential profitability.  Which reminds me, you should really check out The State of the Web, Spring 2012 on The Oatmeal.

But I digress, the whole Facebook thing has been nauseating and really worth an entry of it's own, but at the heart of the problem is the over-inflated sense of importance of social media in general.  Social media has it's place and people have found ingenious ways to make use of it.  But it's not the end-all, be-all of the evolution of human technology.  And let's not ignore how many relationships and I'm sure even lives have been destroyed with the help of social media.

Are you an organization struggling to figure out how to employ social media?  Maybe you don't have to; maybe there are other tools you should be investing in altogether.  Social media has become the hammer, and now everything is a nail.  As a consultant offering software development services, I've gotten some strange requests related to social media.  The funniest one was an organization who provides services to people with involvement in the criminal justice system asking if I could set up Facebook accounts for all of their clients that could be controlled and accessed by the organization.  Um, no, and more importantly, why on earth would you want to do that?

Oh yeah, and social media is not new.  I'm not sure how people have been convinced of this, but I'm sure everyone involved in the early development of Bulletin Board Systems in the late 1970s and early 80s cringes when they hear how "revolutionary" this all is.

You know it's bad when you forget about chocolate

A week ago I bought myself a dark chocolate bar.  It was very much an impulse buy one morning in the midst of the Client Cutover From Hell.  I slipped that bar of chocolate into my purse... and today I remember it's there.  That chocolate bar that I had a week ago decided might give some brief chemical reprieve.  And now as I sit at the tail-end of the Client Cutover From Hell, I celebrate with my chocolate.  It really tells you more about this week than much else I could say.  And yet I will.

I have a few gripes after going through this process.  These are things I have long been sick of but now that they've become the cause of much sleep deprivation, I therefore declare war on....

• Crap software companies that over-charge non-profits with promises of technology like you haven't seen since the mid-90s.  I'm looking at you, Convio Luminate!  This I think is worth a post in and of itself.
• Crap IT support who don't have any clue what they're doing. Some of my difficulties ended up being the result of having to reverse engineer what turned out to be a completely asinine domain configuration by this organization's IT support consultant.  They've already expressed dissatisfaction with this guy, and I'm going to recommend that they drop him altogether.
• Visualforce reRenders.  I used to think it was me, that there were just nuances to reRendering in Visualforce that I was missing.  And while I'm sure that is the case at times, I'm now also quite sure that the way reRender works in Visualforce can be just plain inconsistent.  I discovered in this project that pages that I wrote months ago that were working perfectly fine, now had to re-written because reRenders which previously worked no longer did.  My fellow developer and contractor has found the same thing and has taken to using a third-party Javascript to do the reRenders in many scenarious.  I'm with her (yup, another female developer!)

Sweet Breaded Chicken

Necessity is the mother of invention and this recipe came about when I realized I didn't have an egg to stick the breading to my chicken.  I actually prefer this method now.  I use this chicken either in stir fry or as chicken fingers with some fries.  I'm considering incorporating parmesan into the breading and garlic into the soy sauce next time I do this, and may update the recipe if that works out.

Ingredients
1 lb Chicken Cutlets
3 Tablespoons Flour
1 Cup Soy Sauce
3/4 Cup Plain Breadcrumbs
1/2 Cup Canola (or other vegetable) Oil
Pepper to taste

1. First set up your station so you're ready when you're hands are covered with chicken and goo- one bowl with flour, another bowl with the soy sauce and a plate covered in a layer of bread crumbs.
2. Cut the chicken up into either nugget size or finger size pieces.
3. Coat the chicken pieces with flour.
4. Dunk the chicken into the soy sauce but don't leave it too long so you don't lose all of the flour on it, this helps make it sticky for the next step.
5. Press the chicken into bread crumbs so they're completely coated. 
6. Heat up a pan over medium heat with the oil.  I don't deep fry, I just use enough oil to coat the pan and fry one side of the chicken at a time.
7. Don't add the chicken to the pan until the oil is nice and hot or the breading will just end up mush.  The chicken should sizzle a bit when you add it.
8. Pepper the exposed side of the chicken.
9. Leave the chicken alone while one side of it browns- the time will vary based on the size and thickness of the pieces, but the finger-sized pieces I did took about 4-5 minutes per side.
10. The chicken is ready to flip when the side that's frying is verging on dark brown - the sides should look lightly browned. Tongs are the easiest way to flip them.
11. Pepper the newly exposed side.
12. Brown the other side. 
13. Remove the chicken from the pan and let it drain on a plate with a paper towel beneath it.

Random Book and Movie Reviews

Movies
Contagion- I told you if you left Gwyneth Paltrow to her own devices long enough that something like this would happen.  I liked this movie, though it just re-focused me on my fear of the impending plague.  The characters were well written and performed and it kept me engaged, unlike...

Tinker Tailor Soldier Spy- This is a movie of old white men looking at each other awkwardly and staring across rooms and out windows.  Every once in awhile a woman will look across a room with an expectant dismay on her face.  The men never seem to expect anything until it's too late and unfortunately it's on them who we focus.  The time jumps were confusing and each scene I had to check for grey hairs to get an idea of when it was.  There was a slew of scenes with a character who had been shot earlier that I assumed were flashbacks but it turned out he just hadn't died. I guess Gary Oldman delivered the 10 or so lines he had really well, but mostly he just stared.  If you couldn't tell already, I was not as enamored with this movie as the critics.

Rise of the Planet of the Apes- The writers spent time endearing the apes to me, but there was no need; I was with them from the beginning.  Get 'em, gorilla!  Along those same lines, we were flipping through the channels the other day and happened upon a scene at a bull fight of a bull jumping into the stands of people who had come to watch him be tortured.  Havoc was wreaked. Fair play to the bull. 

Book
Pym, A Novel by Mat Johnson- It's hard for a book that goes into the realm of complete implausibility to fully engage me the way this one did.  The main character, formerly a professor of African American studies, becomes convinced that the novel The Narrative of Arthur Gordon Pym of Nantucket, by Edgar Allen Poe is actually based on a true story and sets off to prove it, an adventure that eventually lands him in Antartica.  It's a very entertaining read, I recommend it.

Visualforce to Word

I've been having lots of fights with Microsoft applications lately.  Perhaps as a web developer, this goes without saying.  I just won a fight getting a Visualforce page to display correctly when converted to a Word file.  I was generating a file of certificates with each certificate on a separate page and a very specific layout that had to be followed. Thought I'd share a few tips on what I learned.

Basics
First off, how to get a Visualforce page to generate as a Word file.  In the "page" tag, include the following:

<apex:page contentType="application/msword#FileName.doc" showHeader="false" standardStylesheets="false">

Page Formatting

• Save a Word doc as HTML
  Not sure how to get the font, border, layout, etc specified in Visualforce to display correctly in Word?  Do it in Word and then save it as an HTML file.  The file it generates is a bit of a beast, but if you scroll down, you'll find <style> tags with Word-specific attributes you can use in <style> tags in Visualforce. You can also copy and paste paragraphs from the html into your Visualforce, though I wouldn't recommend trying to paste the whole file, there's just too much crap (very technical term) in there and Salesforce won't even let you save it the way Word writes the html tags.
  
  For my document, I grabbed these styles to create the following in Visualforce; note the orientation style to set the page as landscape:
  <style>
          @page Section1{
              size:11.0in 8.5in;
              mso-page-orientation:landscape;
              margin:.25in .25in .25in .25in ;
              mso-header-margin:.5in;
              mso-footer-margin:.6in;
          }
          div.Section1
          {
               page:Section1;
          }
          body {
              font-family:"Gotham-Book";
          }
  </style>
  For the style above to work, you'll need to surround your content in Visualforce with: 

<div class="Section1">your content</div>

• Use "in" (inches) for widths to layout the page
  Word is designed to render documents for printing, so widths should be specified in inches (and I'm guessing centimeters for everyone that's abandoned our archaic measuring system).  Just a note that "px" does work in Word (in specific contexts) but if you're trying to fit something to a page like a border that goes around everything, save yourself the trouble and just do it in inches.
• Use "pt" for font sizes
  Setting the font size using style="font-size:12pt;" in VF is the equivalent of selecting 12 for the font size in Word.
• Margin instead of Padding Padding wasn't doing the trick to space out lines of text but margin works.  Here's a sample:

<p style="font-size:22.0pt;font-family:Gotham-Medium;margin-top:12pt;margin-bottom:.0001pt;">
                Some text here
</p>
• Page breaks
  Wherever you want a page break, insert this:
  
  <br clear="all" style="mso-special-character:line-break;page-break-before:always;"/>
  
  Note that I tried using "page-break-before:always;" as the style for a div and that was a no go, Word wants it in a break or paragraph tag.

Including Images

• Don't use a secure URL (https) as the image URL
  I came across forum posts where people were able to successfully use the secure Salesforce URL (i.e. https://na2.salesforce.com/...) to link to images in Word, so my guess is that this varies based on the version of Word you have.  I have Word 2007 and the images would not display until I switched to a non-secure URL, which I did using Salesforce Sites.  It's probably a good idea to do this even if the secure URL works for you because it may not work for others.  
• Use the full URL
  If your image is stored in Salesforce Documents, you can generate that URL like this:
  http://yourdomain.force.com/servlet/servlet.ImageServer?id=015D0000000Dpwc&oid=00DD0000000FJbGyourdomain = 
  Your Salesforce Sites domain
  id= The Id of the Document
  oid= Your Organization Id, which you can find in Setup in Company Information
  
  You can also store the image as a Static Resource and link to it that way.

Exclude "Visualforce" from your web searches

Still need help formatting your Visualforce page for Word? When it comes to figuring out formatting in Word, Google (as is often the case) is your friend and no doubt how you arrived here, but don't limit your searches by including "Visualforce" in the search terms. People use html to generate Word files in other languages, and some of the best tips I found were on PHP and ASP forums.

African Groundnut Stew

I somewhat randomly selected this as our New Year's Eve dinner, and was happy I did.  This is based on a recipe from a slow cooker pamphlet I no longer have, so I can't properly attribute it.  Nor do I know for sure if it's actually based on an African recipe as was claimed.  Whatever, it's yummy.

Ingredients
1/2 to 3/4 lbs Chicken Cutlet

1 Medium Potato
1 Medium Sweet Potato
1/4 Medium Onion
1 Tablespoon Peanut Butter
1 Tablespoon Tomato Paste
1 teaspoon Ground Ginger (or dice up some fresh ginger)
Salt and pepper to taste

1. Cut up the chicken, potatoes and onion into stew sized pieces.  Make the sweet potato pieces bigger than the regular potato pieces as they cook through faster.
2. Mix all ingredients into the slow cooker.
3. Turn on the slow cooker (I have a rather simple unit that's either on or off, but if you need a temperature setting, I'm guessing it will be about median of your options).
4. Let it cook at least 2 1/2 hours, stirring occasionally.

Salesforce general sentiments

Right now my business is based almost exclusively on developing on the Salesforce.com platform, but I'm not one of those Marc Benioff idolizing, Salesforce can do no wrong type of people (those people are annoying).  I think Salesforce has done a good job of creating an extensible CRM that allows for non-technical people to customize it up to a point, and developers to customize it beyond that point.

For awhile, the supposed plan for Salesforce was to become a true platform- they handle the storage and maintenance and provide the framework on which everyone could build or install pre-built applications.  Included in that pitch was the statement that they wouldn't build enhancements specific to industry verticals or job functions, that would be left to the customers and third-party developers.  Good bye Salesforce.com, hello Force.com.  I liked this plan. 

This is clearly no longer the plan.  Salesforce continues to build functionality for sales and other verticals, and has gone the Microsofty route of acquiring companies who have built Salesforce solutions and incorporating those solutions into the platform.  This plan I don't like for the same reason I never cared for Microsoft doing it.  Trying to incorporate software into a platform that was not specifically engineered to be incorporated creates complexities (i.e. opportunities for bugs) and inefficiencies that could have been avoided by engineering something from the ground up to actually be a part of the platform.  It's a big part (though not the only reason) of why Windows is the giant bloated piece of crap that is today.  Microsoft has managed to prosper despite this by having a strong foothold in the market, particularly corporate sector of the market, and because computers have progressed to be able to handle the bloatiness of Windows and still function. 

As of late, we've seen some instability in the Salesforce sandbox environment. There was that week this past fall where one of the sandboxes was dead to the world, and just last night the entire sandbox environment, in North America anyway, was hiccuping.  Do I know for sure that these glitches were the result of the way they've expanded the platform?  Absolutely not.  But they were no real surprise to me when they hit and I suspect we'll see more; it's as hard for Salesforce to avoid as a painfully long boot-up time is for MS Windows.  Be careful there Icarus, your wings are looking a little gooey.

Zoo Life - Patricio

Each week I volunteer half a day at the Prospect Park Zoo.  Anyone familiar with my background won't be surprised by this.  Spending time regularly at the zoo provides a unique opportunity to become familiar with the animals in ways that occasional visitors can't.  The amount of time alone makes it more likely to witness telling incidents, and some animals will become familiar with you and interact in ways they won't with strangers.

I suspect that any volunteer at the PPZ has stories related to Patricio, the Cape Barren Goose.  Patricio resides in the Australian walkabout, which means he's free to roam about with zoo visitors along with the wallabies and kangaroo.  Well, usually anyway.  Unlike his would-be mate Ethel who keeps herself hidden most of the time, Patricio spends his day right on the walking trail, usually at the side of the zoo employee or volunteer charged with minding the area.  Just about the only time we see Ethel is when she comes out to give Patricio a piece of her mind when he's acting up and honking.  Ethel is the only one Patricio defers to, and he'll skitter away nervously when she comes running at him.

Patricio

Some days when I'm with him, Patricio will stay right by my side, reaching out to nip at my coat or shoelaces occasionally when I'm not looking.  When he's in a mood, he may decide he'd rather run around with a group of schoolchildren.  Ever see a child panick in fear that they're being chased by a wild goose?  Nothing excites Patricio more- to him it's just a part of the game, while children squeal in terror that they're about to be mauled.  The walkabout has a double-door system to enter, and signs warn visitors to keep only one door open at a time.  This did not stop Patricio from wandering out behind a group of school children one day, following them down the path towards the porcupines before anyone took notice and alerted staff.

Patricio Sneaks Up on Me

Such incidents prompted the animal keepers to relegate Patricio to a fenced in area at the back of the walkabout for a few weeks this fall.  To Patricio, this was jail.  He did not get along with his fellow residents, a noisy flock of guinea fowl and I often saw him charging angrily at them or a squirrel who had wandered in for some free zoo food.  Other times I'd see him practicing his escape.  He'd stretch his wings out to his sides, look in front of himself determinedly and waddle forward as quickly as he could, certain that this would be the time he'd break into flight (I believe his wings are clipped).  It's a bittersweet episode to witness.

I was happy when Patricio was released back into the walkabout proper.  The wallabies are adorable and interesting to watch, but spend most of their time hidden as does the kangaroo, who in her old age is more interested in finding a quiet place to nap.  Patricio is a handful, but in the end he's harmless and he keeps things interesting.

How did I miss Moonraker?

An aside- according to Michele Bachman in the debate last night, we need to "legalize American energy".  Here, here!

How have I not seen Moonraker yet?  This shall be resolved tonight.  In the cottage my family rented over the summer, there was a poster for this movie and I hadn't even realize it existed.  From a review: "My favorite part: when the astronauts get killed in this zero-G environment, they sometimes start falling, as if their willpower alone had been holding them up." He wrestles a snake in the water, has a speed boat chase with gunfire, the excitement seemingly never ends in this epic trailer:

Which reminds me, I completely forgot "Bond Villains" from my list of human(oids) for whose demise I cheer.  Reactions from viewing this movie to come...

Update: Now that I've seen it, I really can't believe I missed that movie, it is a classic.  Don't get me wrong, the plot was disjointed and the dialogue often ridiculous (hilariously so).  But really it gave me exactly what I want from a Bond movie- a series of implausible adventures seeking/fighting with over-the-top villains.  There was not one but two high speed boat chases, one in Venice which ended with Bond showing a complete disregard for public safety by speeding through crowded plazas in a gondola equipped with a hovercraft skirt.  There were sky diving fight scenes, aerial tramway fights, the fight with a giant snake, fights in space, the excitement was just barely broken up with quippy comments between Bond and sexy women and Bond and giant-headed villains.  Oh, and his main squeeze in the flick is actually named Dr. Goodhead.  Yeah, seriously.  If you haven't seen it, or haven't seen it since you were a kid, I highly recommend it.

Simile Timeline in Salesforce, Revisited

The Simile Timeline is a javascript widget that's available as open source to create interactive timelines.  If you go to the main page for the Simile Timeline, you'll see how you can drag the timeline back and forth through time and click on items to see a detailed description; it's pretty slick.

Back in another life, I had implemented this timeline in Salesforce to provide a graphical display of a Contact's history on their detail page: Events, Tasks, (Job) Opportunities, Interviews, Jobs.  I did it by customizing the free Timeline S-control mashup that utilizes that Simile Timeline widget.  It provided a nice graphical narrative that you could scroll forward or backward in time through, and clicking an individual item on the Timeline would display details on and a link to the record.  This is the screenshot of that application from the appExchange:

That application is still available, but in it's original S-control form, now replaced by Visualforce.  Salesforce still supports S-controls, so you can  use it if you're fine with the default configuration it comes with.  If you're going to need to customize the information that's displayed on the timeline, for instance showing custom object child records to Account, Contact, Case or Opportunity or values of custom fields in the pop-up details, I'd recommend just doing this in Visualforce, which is what I've just done for a client.  Anyone with a bit of javascript and Apex savvy can easily implement this themselves.

I'm considering creating a nifty little package that anyone can install to implement it with no code, probably with an admin screen to let Sys Admins select the Timeline settings such as the intervals of time displayed, and a query interface to display custom object records or fields without requiring any code.  Interested?  Leave me a comment.

Not being an asshole wins the race...

Having a tortoise as a pet, I often think of the Tortoise and the Hare fable.  The sum up of that story is "slow and steady wins the race".  The tortoise didn't win because he was slow and steady, he won because that hare was an asshole.  To me, that's a much more important lesson than slow and steady- being an asshole can bite you in the ass.  And that's exactly how you should explain that to your children.  I only wish that were a more predictable outcome. 

My tortoise is very cute; he even likes to snuggle with me-

Still don't think he's cute?  How about this?

What I've really learned from Rokku is determination.  That boy will not give up.  When I take him out on the terrace, 7 flights up, he sometimes gets it into his head that he would like climb over the edge and see what's out there.  I've tried to explain to him that that would be suicide.  I pick him up and move him away, he scrambles right back.  The more you pull him away from his perceived destination, the harder he will try to get there.  Rokku, stop burrowing into those electrical cables!  It's admirable if not nerve-wracking, and I love him for it.

Oh how I loves me some zombie violence

I've discussed this on Friendface and with friends, but figured I'd include my thoughts on it here as an insight into the minds of people.  I was originally going to say it was an insight into my mind, but this is one of those commonalities that brings us together as a species.

There are certain movies and books with scenes of brutal violence against humans (or humanoids) to which we are very much drawn, and zombies stories are a prime example.  I'm also always up for some Nasi violence too.  Our lizard brains get a thrill from violence, it's deep within us from millions of years of  evolution as meat eaters and territorial beasts.  But our rational minds tell us that we can't just go around slaying and beating at will (well, some of us anyway).

Zombie and Nasi violence presents us with the rare opportunity to enjoy a healthy dose of violence with the moral quandry removed.  Storytellers have long been aware of this; it's why they go to lengths to build up the evil qualities of a character before presenting us with their gruesome death.  The more they've worked to show you just how evil someone is, the more painful, gory detail you can expect in their last moments.

Here's a list of some human(oid)s for whose demise I will cheer (did I un-dangle that participle correctly?).  I'll admit that it gets a little ethically questionable towards the end.

• Zombies
• Nasis (becoming an over-used antagonist in movies again)
• Khmer Rouge party leaders (an under-used antagonist in movies that I vote to now replace Nasis)
• Aliens plotting the destruction of our planet, as implausible a concept as I find this to be
• Conquistadors
• Serial killers (or single instance killers, depending on the circumstances)
• Child rapists
• Oh, any rapists really
• Followers of Sauron
• Software agents posing as humans in a virtual reality created to control humans and use us as batteries, as implausible a concept as I find that to be
• Anyone who shouts "Death to America!" (hey, I'll be the first to admit our imperfections, but fuck you, man)
• Anyone Dick Cheney has ever trusted to do his dirty work
• Dick Cheney
• Hedge fund managers
• Executives at companies that implement hydrofracking (I've been a little fixated on this lately.  I like water)

And let's be clear, I'm not speaking about this in terms of death penalty legislation, on which I have a completely different stance, or sanctioning rampant murder sprees of people who fit these descriptions, which would put you squarely in the serial killers category.  This is purely regarding the mutilation and death of fictional characters for my entertainment.  But nonetheless, I accept the impact this will have any future runs for office.

Why won't this damn thing reRender properly...

Since Visualforce came into being, the aspect of it I've had the most trouble with is the reRender, or more specifically, getting reRenders to behave the way I want them to.  This is partly because much of the coding I did before Salesforce was behind the scenes logic and not web interface design, but it's partly because reRenders can be a real pain in the ass.  Along the way, I've figured out some tips and thought I'd share.

actionRegion
You will drive yourself insane with reRenders if you don't know how to use actionRegions. actionRegions are to submitting what reRenders are to rerendering.  Perhaps a little more explanation and a demo are in order.

When you perform any action on your VF page, by default the entire form is submitted.  You may not actually need everything submitted though; you may just be need a portion of your variables submitted, or maybe none at all.  Submitting everything needlessly means that not only are you slowing down performance of your page, but the page is attempting to validate what's being submitted before it sends anything back to the controller.  So if you have any required inputFields, for instance, those fields will need to be filled in before the form submits at all.  And if you don't know that's what's happening, either because your form has no <apex:pageMessages/> or it's not being reRendered, it will appear as if your action isn't working for no reason at all. 

Let's use this simple page and controller as an example, first without an actionRegion (incorrect):

Page before adding actionRegion (the wrong way)

<apex:page standardController="Account" extensions="extTest">
    <apex:form >
        <apex:pageBlock>
            <apex:pageBlockButtons >
                <apex:commandButton value="Show Billing Street" reRender="opBilling">
                    <apex:param value="true" assignTo="{!ShowBilling}"/>
                </apex:commandButton>
            </apex:pageBlockButtons>
           
            <apex:pageBlockSection >
                <apex:inputField value="{!Account.Name}"/>
            </apex:pageBlockSection>
           
            <apex:outputPanel id="opBilling">
                <apex:pageBlockSection rendered="{!ShowBilling}" id="pbMailing">
                    <apex:inputField value="{!Account.BillingStreet}"/>
                </apex:pageBlockSection>
            </apex:outputPanel>
        </apex:pageBlock>
    </apex:form>
</apex:page>

 Extension:

public with sharing class extTest {
    public Boolean ShowBilling {get;set;}
   
    public extTest(ApexPages.StandardController controller) {
        ShowBilling = false;
    }

}

The idea is, the page opens showing only the Account Name field. Clicking the Showing Billing Street button will update the ShowingBilling variable to true and reRender the opBilling outpanel so the Billing Street field is displayed.  As it's written above however, it will not work if the Account Name field isn't filled in first.  That Name field is required, so the command button tries to submit the whole form and it fails validation without a Name value entered.  Since only the opBilling outputPanel is being reRendered, you won't event see an error message since pageMessages isn't being reRendered (Debug logs are your friend when troubleshooting such issues).

One way to fix this is by updating the page with an actionRegion around the commandButton like so:

pageBlockButtons after adding actionRegion (one right way)

<apex:pageBlockButtons >
                <apex:actionRegion >
                    <apex:commandButton value="Show Billing Street" reRender="opBilling">
                        <apex:param value="true" assignTo="{!ShowBilling}"/>
                    </apex:commandButton>
                </apex:actionRegion>
 </apex:pageBlockButtons>

Now with actionRegion around that button, only the values inside of the actionRegion are submitted, so BillingStreet will be rendered with or without a value in the Account Name field.

Immediate
For the sake of keeping the example above simple, the button didn't do anything other than updating ShowBilling using the param tag and reRendering opBilling.  In reality, you'd only need actionRegion when you need some portion of the variables on the page submitted to the controller/extension to perform logic in a method.  If you don't need to submit any variables, then you can use the "immediate" property of the tag that's performing an action.  Let's see this with the example above:

pageBlockButtons after adding immediate (another right way)

<apex:pageBlockButtons >
                <apex:commandButton value="Show Billing Street" immediate="true" reRender="opBilling">
                    <apex:param value="true" assignTo="{!ShowBilling}"/>
                </apex:commandButton>

</apex:pageBlockButtons>

Make sure you know what immediate means before using it. I've seen people set immediate to true to avoid issues with required fields not realizing that their controller/extension was not receiving any updated variable values from their page, thereby screwing up their logic and leaving them flummoxed.

Choosing what to reRender
I'll probably expand on this in a separate post because there are a lot of scenarios to consider, but I'll use the example above to give one basic tip. 
You may have noticed in my example that I've surrounded the pageBlockSection that contains BillingStreet with an outputPanel, and the reRender acts on that outputPanel.  This may seem like an extra component- why not just reRender the pageBlockSection itself and skip that outputPanel altogether?

What I've learned is that if a reRender updates a variable on your page, you need to reRender a parent of the tag that contains that variable, and not the tag itself. OutputPanels are a useful means of doing so.  To see this in action, try updating the example above so the button attempts to reRender the pageBlockSection directly:

pageBlockButtons with reRender of the pageBlockSection (wrong way)

<apex:pageBlockButtons >
                <apex:commandButton value="Show Billing Street" immediate="true" reRender="pbMailing">
                    <apex:param value="true" assignTo="{!ShowBilling}"/>
                </apex:commandButton>
  </apex:pageBlockButtons>

If you try this example with the commandButton set to reRender the pageBlockSection, it just doesn't work.  So either you need that outputPanel surrounding the pageBlockSection, or alternatively you could move the rendered="ShowBilling" property to the inputField instead, but in this example, that would mean you're rendering an empty pageBlockSection.

As above, there are more scenarios to explore here, but I'll leave it at this for now and hopefully this is helpful to people frustrated with the nuances of reRendering.