Use reference tables for previews in Power BI

In my quest to provide customers with awesome reports I’ve been looking at why the performance in Power BI can be so slow at times. As I pointed out in earlier posts the OData.Feed can be quite the resource hog depending on the endpoint, so I’ve started to fall back on Json.Document to get much better preview times (but not without it’s drawbacks, like record count, connected tables and option set labels).

Just because I’m kinda new to this I like to perform a lot of the logic using M in the query editor, then I sew it all together with relationships and measures inside the reports I build. I noticed that the previews kept getting slower the more I logic I added, so get the ultimate magic tool and looked at what was happening.

First step: Json.Document(Web.Contents(“https://<org>.api.crm4.dynamics.com/api/data/v8.2/opportunities”))

OK, so it makes a request to Dynamics and retrieves the data in 3.435 seconds. That’s not fantastic, but it’s not bad.

Second step: = Source[value]

OK, no requests made. That’s what I would expect from it

Third step: = Table.FromList(value, Splitter.SplitByNothing(), null, null, ExtraValues.Error)

Still no requests made, that’s great

Fourth step: = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"actualclosedate", "actualvalue", "budgetamount", "estimatedclosedate", "estimatedvalue", "name", "opportunityid", "pzl_countryregion", "statuscode"}, {"Column1.actualclosedate", "Column1.actualvalue", "Column1.budgetamount", "Column1.estimatedclosedate", "Column1.estimatedvalue", "Column1.name", "Column1.opportunityid", "Column1.pzl_countryregion", "Column1.statuscode"})

Guess what? Still no requests.

Fifth step: = Table.TransformColumnTypes(#"Expanded Column1",{{"Column1.estimatedclosedate", type date}})

No requests. Great, so we only make 1 request per step, as long as we don’t merge in more columns? Not quite.

Refresh preview

Wait what? 5 requests, that’s 1 request per step. That makes the loading time increase from the original 3+ seconds into well over 15 seconds. I don’t have the answer for why it makes a new request for each and every step, but let me show you the consequence of this when you use OData.Feed instead of Json.Document

These are the steps taken:

1. = OData.Feed(https://<org>.api.crm4.dynamics.com/api/data/v8.2/opportunities)
2. = Table.SelectColumns(Source,{"actualclosedate", "actualvalue", "budgetamount", "createdon", "estimatedclosedate", "estimatedvalue", "name", "new_calculatedrevenue", "new_product", "opportunityid", "pzl_countryregion"})
3. = Table.TransformColumnTypes(#"Removed Other Columns",{{"createdon", type datetime}})

Now that should be enough to make you think twice before reporting on Dynamics data in Power BI. As you can see it’s the metadata documents which takes the longest, but for some reason there are 4 requests to the “all opportunities” endpoint and the metadata endpoint, while there are 3 requests to the opportunity endpoint with reduced number of columns (fields). Also, there is one top=1000 request which completes before the last two. This tells me that when you refresh the data, the requests are made in sequence for each step, but the steps are performed in parallel. It almost smells like premature optimization.

So let’s break this down and do it step by step.

Step 1

OK, great, one request for the opportunities and one for the metadata, finally one to get the top 1000 records for the preview.

Step 2

Now we see that it makes the same three requests as it did in the previous step, but it also performs another set of requests against “all opportunities” and metadata before it performs a request against opportunities with reduced number of columns (fields).

Step 3

Finally, it does all the requests from previous steps, but it also adds another duplicate for this current step (which only converts the type, and doesn’t require collecting more data from the source system).

The magic workaround

Just by pure luck, when I tested this originally I got 4 duplicates of the “all opportunities”+select requests, and I couldn’t figure out where the 4th duplicate came in. Turns out, because I made a reference from this opportunity query into a new query, and the data is refreshed in all referenced queries as well as the parent(s).

That made me test one final thing. I created a new reference query after I had done all my logic in the main query.

Then I refreshed the data while in the reference query.

Step 1: = Query1

Voilà! Only one set of requests, even though it’s technically the fourth step.

This means I just saved myself many duplicate requests, and the data loads much faster than for each subsequent step. This was only for 3 steps, imagine how much this will matter if you have 10, or 20 steps in your queries.

The wrap up

So in summary, when you refresh the preview for a query in a Power BI, all the requests are are repeated and added on for each step you have in that query. It’s an exponential growth of data gathering, and can make everything slow as nothing else.

Creating new queries with references removes the need to duplicate all the requests, and you can keep working with and massaging the data without having to worry about minutes of data reloading.

Using related entities in PowerApps with MSDYN365 as a data source

Getting related records to work in PowerApps and MSDYN365

PowerApps is a relatively new feature to the Office365 portfolio which allows advanced users to create their own apps which can be used on mobile (and desktop) platforms to perform daily tasks. It is a great tool for creating those small functions you usually have to book a developer for a few hours/days, and is ready for authentication, authorization, and distribution right out of the box.

PowerApps comes with the ability to connect to a wide array of sources to collect and manipulate data, and today I'm going to demonstrate how to connect with MSDYN365:CRM and how to use connect with related records.

Connecting to MSDYN365:CRM

Now this is the easiest part. To connect with MSDYN365 in your powerapp simply go to the 'Content' tab and select Data Sources, and you'll be presented with the option to add new connections and data sources. Here it will probably show you something like this (I have a completely new trial organization with only O365 E3 and MSDYN365 Plan1:

What I'm doing is connecting to the MSDYN365 suggestion, and I'm going to add 3 tables (entities) to my data source list; Accounts, Contacts and Users (systemusers).
Once that's done I'm going to add a new screen with the browse template. This will give me the option to search and display records from the chosen data source.
Now, from the right hand side, choose the advanced tab and select your Gallery component from the dropdown list. Next through the Home tab on the top toolbar select "Items" from the property dropdown box. We're going to do some drastic changes to the formula used to collect data, but first a little insight.

When a function is run/validated in powerapps it returns the result set to the parent function (or context, whatever you'd like to call it). This means that we can build an advanced data source or query by nesting functions inside each other, and end up with a pretty good set of data to play with.

So to get the data I want in my gallery browser I'm going to use the following formula:
SortByColumns(Search(AddColumns(Accounts, "User", LookUp(Users, systemuserid = _ownerid_value), "Contact", LookUp(Contacts, contactid = _primarycontactid_value)), TextSearchBox1.Text, "accountnumber", "name"), "name", If(SortDescending1, Descending, Ascending))

Now this might seem a little frightening at first, but I'll go through it step by step and you'll see how I thought through it. Remember the part about nesting functions that I mentioned above? Well that's how I start building a function, by nesting backwards.

1. I want a collection of accounts, but I also want to add the primary contact and the account owner (I'm not checking for teams, I'm just assuming that all accounts are owned by users. This is just a quick demo). That means I'll have to use a function to join those sources. A quick search online says that I can use AddColumns to do joins in a PowerApp formula, and thanks to the built in intellisense it's easy to create the formula required.
   AddColumns(<data source>, <table alias>, LookUp(<data source>, <condition>))
   This will nest the related table underneath the table alias (or namespace if you will), and you can add additional tables by just specifying new column aliases and lookup expressions. To get the ones I needed I filled in the following:
   AddColumns(Accounts, "User", LookUp(Users, systemuserid = _ownerid_value), "Contact", LookUp(Contacts, contactid = _primarycontactid_value))
   This means that I've nested the Users entity underneath the "User" column alias, and the Contacts entity underneath the "Contact" column alias.
2. I need to be able to search for these records, I don't simply want all the accounts listed at all times. Search is a function designed for this, but I could also use Filter if it was a set query I wanted to use. Now I know that the AddColumns(...) functon I used will return the result set of that function, so I could potentially search through the related records as well. I'm going to do it a lot simpler so I'll want to search through the account number and name field on account. The syntax is :
   Search(<data source>, <Search text>, <Column to search in>, <Column2 to search in>, ...)
   The data source in our case will be our previous AddColumns formula. The search text we'll get from the default search box, TextSearchBox1.Text, and the columns we want are "accountnumber" and "name", which makes the formula look like this:
   Search(AddColumns(.....), TextSearchBox1.Text, "accountnumber", "name")
3. Lastly I'd like to sort the result, so I'm going to add a SortByColumns function just to get a predictive list instead of a random one. This starts just the same as our previous function, the result of everything we've done so far has been returned, and we can order by the columns we want. The syntax looks like this:
   SortByColumns(<data source>, <column name>, If(<logical test>, <true value>, <false value>))
   We want to use the data source we've built so far, and we want to use the "name" column of the account. The logical test will be the default sorting button, with true as Descending and false as Ascending (false is the default value). We end up with the following formula:
   SortByColumns(Search(...), "name", If(SortDescending1, Descending, Ascending))

Now we've got a search gallery were we can search for accounts and get the related primary contact and the owning user data values. We can use that to further style our result list so it will look something like this:

Customizing the details form

So now that we have a decent-ish looking gallery we want to do some modifications to our details and edit forms. There isn't really any magic to this, you simply add and remove the data fields that you want. There are however two things I would like to point out so that you don't walk into the sweet honey traps.

First of all, we want to set a data source and item that PowerApps has a relation to and can work with. If we choose the same data source as we used in the gallery then we can certainly we the data and place it on the form, but updating the record or creating a new one could prove a bigger challenge. So what we do here is that for the data source, we set the "Accounts" entity table. The Items property is a bit more special though, and here we have two options (I'm going for option 2):

1. We use the item from the gallery, ie. BrowseGallery1.Selected. This will work perfectly fine as long as you don't try to update any other fields than the main account fields. However, you will get an error in the PowerApp user interface because the data source chosen for the details form is not the same data source as you have in your gallery. The upside is that you don't have to do a new query to get the account data, but on the downside you will have to live with a warning sign that comes with not following the best practices.
2. We use the accountid from the gallery to look up the specific account from the account data source, ie we use the following formula:
   LookUp(Accounts, accountid = BrowseGallery1.Selected.accountid)
   The positive thing about this is that we're now 100% compliant with the best practices, but we now don't have direct access to the sub-components, and we're doing an additional query to CRM for a lookup. Luckily, this lookup is very cheap because we're getting 1 entity based on the Guid, and I'll be picking this one just so we can demonstrate how to do a lookup inside a field.

So now that we've got the data available we can start adding the fields to the forms. If we want to have some fields from related records we can simply use the LookUp function to get them. For example, if I want the full name of the primary contact I can use this formula in a data form:

LookUp(Contacts, contactid = _primarycontactid_value).fullname

So now that I've done a very small amount of customization my detail form looks like this.

Customizing the details form and changing the primary contact

So now we're ready to do some customization to the edit form. I've gone ahead and added the same data source and item as we did on the detail form, and I've added the address fields just so we have something to play with. The results so far looks like this:

Now, as a power user I would love to be able to edit the primary contact of the customer without having to specify a Guid, I want a search window so that I can find the right one and add it to the account I'm looking at. The first thing I'll do is to add a new browse gallery screen, and then I add a new button on the edit form which navigates to the new screen:

Now, on the contactselector screen I want to show all active contacts. I'll start off by creating the same kind of formula as we did in the beginning of this blog post, but this time I'm just searching through the Contacts data source. My formula now looks like this:

SortByColumns(Search(Filter(Contacts, statecode = 0), TextSearchBox2.Text, "fullname"), "fullname", If(SortDescending1, SortOrder.Descending, SortOrder.Ascending))

Now the right arrow on each result will usually try to navigate to a details window, but we don't have that window here and we don't want to see the details either, we just want to add this contact as the primary contact to the account. So what we're going to do is to use the "patch" function in PowerApps. The Patch option allows you to edit items on an existing record, while the update function replaces an old record with a new one. Since we just want to update the contact I'm going ahead with the patch method. The syntax for the patch function is like this:

Patch(<data source>, LookUp(...), { field1: value1, field2: value2})

So we want to update the account we have selected in previous windows. In additon, we want to update the primary contact Id for that account. That means we have to do a formula like so:

Patch(Accounts, LookUp(Accounts, accountid = BrowseGallery1.Selected.accountid), { _primarycontactid_value: ThisItem.contactid})

This is starting to look pretty nice, but we also want to navigate away from this form after updating. To do that we simply add a semicolon after the Patch function, then call the Back() function to go to the previous window. What we end up with is a window that looks something like this:

So now we have a complete app that can search for Accounts, display related record information, and finally edit the fields and set a new primary contact. This should work just as well for all related records, but now you have a pretty good example to start out with.

Wrap-up

So as we've seen not everything in PowerApps is just point-and-click, but you don't have to be an advanced developer to find out this stuff on your own. The documentation for powerapps is decent, so you'll find plenty of examples you can experiment with on your own.

What I find very exciting is that not only can I create these simple apps very fast, I can also distribute them to other users in my organization so that it will be available from the https://home.dynamics.com menu.

I've added a copy of the powerapp to to my public google drive so you can download it and use it as you want (TestAccounts.msapp). 

All you have to do is to go to the "Content" tab and add the data sources, you'll need "Accounts", "Contacts" and "Users", and then you're ready to go.

Errors in CRM for Outlook 2011 when connecting to CRM

I recently had a customer who upgraded their Dynamics CRM 2011 system from update roll-up 11 to roll-up 14, and due to strict policies in the production environment they wanted to test the roll-up 14 for the Outlook client on a pilot group before rolling it out to all users.

After using CRM with the newest rollup for several weeks they suddenly encountered an error where Outlook clients no longer could connect to the servers, and it happened to a lot of users in a short amount of time. When checking the error messages we found that the clients had issues trying to synchronize with the servers (normal synchronization, offline mode disabled), and we found out that they recently imported a new version of their solution during the weekend. All the users who experienced the problem could connect to CRM in their web browser, and a quick check on the servers did not show any errors in the event log. Needless to say this points to an issue with the client.

Proceed to default troubleshooting procedures:
We found a test client we could use, booted up the same image the customer use on their clients with the same version of software and updates, and tried to connect to CRM through Outlook. Like the rest of the users this gave us the following error:

08:56:44|  Error| Exception : Must specify valid information for parsing in the string.    at System.Enum.EnumResult.SetFailure(ParseFailureKind failure, String failureMessageID, Object failureMessageFormatArgument)

   at System.Enum.TryParseEnum(Type enumType, String value, Boolean ignoreCase, EnumResult& parseResult)

   at System.Enum.Parse(Type enumType, String value, Boolean ignoreCase)

   at Microsoft.Crm.Platform.ConvertHelper.EnumFromXmlString(Type enumType, String mask)

   at Microsoft.Crm.Metadata.AttributeDescription.FillPropertiesFromXml(XmlNode node, Boolean throwIfIdsMissing)

   at Microsoft.Crm.Metadata.NonSharableMetadataCacheLoader.<>c__DisplayClass7.<LoadDescriptionsFromXml>b__6(IFillableMetadataDescription description, XmlNode node, Boolean newIterator)

   at Microsoft.Crm.Metadata.NonSharableMetadataCacheLoader.LoadDescriptionsFromXml(String name, MetadataContainer container, CounterList counter, IEnumerable`1 navs, IEnumerable`1 paths, LoadDescriptionFromXmlDelegate LoadDescriptionFromXmlDelegate)

   at Microsoft.Crm.Metadata.NonSharableMetadataCacheLoader.LoadDescriptionsFromXml(String name, MetadataContainer container, CounterList counter, IEnumerable`1 navs, String[] paths)

   at Microsoft.Crm.Metadata.NonSharableMetadataCacheLoader.LoadDescriptionsFromXml(String name, MetadataContainer container, CounterList counter, XPathNavigator nav, String path)

   at Microsoft.Crm.Metadata.NonSharableMetadataCacheLoader.BuildContainerFromXml(XmlDocument xmlDocument, LoadMasks masks, CounterList counter)

   at Microsoft.Crm.Metadata.NonSharableMetadataCacheLoader.LoadCacheFromXml(XmlDocument xmlDocument, LoadMasks masks, CounterList counter)

   at Microsoft.Crm.Metadata.DynamicMetadataCacheFactory.LoadCacheFromWebService(DynamicMetadataCacheLoader loader, LoadMasks masks, IOrganizationContext context, Boolean writeCacheToLocalFile, CounterList counter)

   at Microsoft.Crm.Metadata.DynamicMetadataCacheFactory.LoadCacheForRichClient(LoadMethod loadMethod, DynamicMetadataCacheLoader loader, LoadMasks masks, IOrganizationContext context, CounterList counter)

   at Microsoft.Crm.Metadata.DynamicMetadataCacheFactory.LoadMetadataCache(LoadMethod method, CacheType type, IOrganizationContext context)

   at Microsoft.Crm.Metadata.MetadataCache.LoadCache(IOrganizationContext context, Boolean fileOnlyIfExists)

   at Microsoft.Crm.Metadata.MetadataCache.GetInstance(IOrganizationContext context)

   at Microsoft.Crm.Application.Outlook.Config.OutlookConfigurator.InitializeMapiStoreForFirstTime()

   at Microsoft.Crm.Application.Outlook.Config.OutlookConfigurator.Configure(IProgressEventHandler progressEventHandler)

   at Microsoft.Crm.Application.Outlook.Config.ConfigEngine.Configure(Object stateInfo)

At this point we decided to turn on tracing and collected the error from the trace log as well (English and Norwegian error message):

>Error occurred while refreshing registry cache, cache has been cleared. Exception: System.IO.IOException: Det er ikke mer data tilgjengelig.

>Error occurred while refreshing registry cache, cache has been cleared. Exception: System.IO.IOException: No more data is available

These error messages made me suspect that the issue was content in the new solution which was not supported by the older roll-up version on the client, so we updated the client to update rollup 14 and tried again. As suspected this fixed the issues and allowed the client to successfully connect and synchronize with the servers.

At this point I would like to say that we informed our customer about the dangers of running different versions of the software on the server and client, and we highly recommended installing the updated client on all computers as soon as possible. However, after testing this in their staging environment without any errors they decided to proceed with a prolonged pilot group in production.

On the positive side, update roll-up 12 (and newer) includes some great improvements in the Outlook client which makes it a less chatty, so the users will be able to enjoy a faster, smoother CRM experience with the newest roll-up on both the servers and the client.