"Failed to retrieve the Schema" error configuring IBM DB2 Connector on FIM 2010 R2

We had an issue recently in configuring a IBM DB2 Universal Database Connector which was connecting to a DB Linux server instance.

We followed all the best practices in install the Synchronization Engine Server:

• We locked down Sync Engine Service account.
• We installed the IBM DB2 OLEDB v9.7 FP1 client software.
• We setup a connection to the DB2 server using the OLEDB tools.

While trying to configure the Connector we where continually getting "Failed to retrieve the Schema" and "Cannot Connect to the Database" errors even though we were able to connect to the server using the connectivity tools provided with the OLEDB client. I then monitored the traffic between the two servers using netmon and noticed that the MA was not generating a call to the DB2 Server, but the connectivity tests were.

It seemed that due to the Sync Engine service account lock down, it did not have permissions to access the DB2 Connectivity driver location. Adding the service account to the DB2ADMINS group on the FIM Server. Voila, connectivity was flowing...

Object Creation Limit Reached In Office 365 for Education

While busy with a large deployment of Office 365 for Education (150k users) at one of my clients I ran into this error after the initial 50000 users were created successfully:

You have reached the limit for the number of objects your company can create.

 This happend in both my testing and production domains.

 

I also experienced the following response from PowerShell:

 

Unable to add this user because the total number of allowed objects has already been reached

 

As per http://support.microsoft.com/kb/2812409 these are DS quota limits that are enforced on all Office 365 tenants. I had to create a service request for each tenant with Microsoft to get the limit increased. For more information on how to request the DS quota increase click to  http://support.microsoft.com/kb/2812409

A look at why using Multi-value tables are not always the best approach

I recently had a case at a client where they were unable to get a full import of their user information from Oracle into FIM 2010 with the full import failing after 36 hours without importing all the necessary information. The institution has a 250000+ user base and complex infrastructure.  The Oracle MA utilized a multi-value table for allocation of some RBAC components within their AD.

The Oracle MA had the following tables configured:

1.       Main User information table (250000+ items, 18 fields, 790mb)

2.       Multi-value table (250000+ items, 4 fields,  690mb)

3.       Delta table

I followed the usual troubleshooting steps ensuring that:

1.       The Oracle tables were correctly built and indexed.

2.       The Oracle client version was correct.

3.       The FIM Server was installed, configured and patched correctly.

4.       All possible bottlenecks were identified and eliminated between servers in question.

With all these resolved, we did not get a significant increase in speed, so I did a test of importing the user information excluding the multi-value table from the configuration and the import completed in less than 40 minutes. It was in fact the multi-value table causing the import performance degradation. Although it was correctly indexed, contained only 4 fields, it was almost the same size as the main user table.

We had three options:

1.       Reduce the complexity of the data in the main and multi-value tables (was not possible in this instance).

2.       Consolidate the data into a single view and pass the multi-value processing to FIM rules extension.

3.       Write a custom management agent to incorporate multi-values directly (Due to the work involved in achieving this it was not really an option).

I then asked the client to consolidate the data into a single view writing the multi-values delimited to a single value field. I was then able to still do the import in less than 40 minutes and use a rules extension to extract the multi-values from the single value table to an array which I then flowed to the relevant multi-value attributes in the metaverse. The total process of full import and sync then completed in less than 3 hours.

I am a big advocate of using multi-value tables where appropriate, but there certain scenarios where using multi-value tables are just not feasible and other approaches may be needed.

When evaluating whether or not to use a multi-value table it is important remember that the size and data complexity if the table will affect performance. My own rule of thumb is once the multi-value table exceeds 33-50% (based on complexity) of size of the main table there will be a performance impact on the import of data into the connector space.

FIM 2010 R2 RTM now available on MSDN

As of the 1st of June 2012 FIM 2010 R2 RTM has been released on MSDN And TechNet. What is very notable is that BHOLD is also available.

The media can be downloaded via the following links below:
FIM 2010 R2 RTM: https://msdn.microsoft.com/en-US/subscriptions/securedownloads/#FileId=49037
BHOLD: https://msdn.microsoft.com/en-US/subscriptions/securedownloads/#FileId=49036

For additional information on FIM 2010 R2 please refer to the release  notes

Controlling request retension periods with Initial loading in FIM 2010

In the FIM 2010 deployments involving the FIM Portal we have a period of initial loading of data from the various data sources we are integrating and we often have to load in excess of 150000+ objects into the Portal. This can be a bit tricky as you need to try and balance speed with the amount of sets, MPR's, workflows and sync rules that are applied to the data when it enters the portal.

Within the portal every change made to any object will be processed via the FIM Portal's workflow engine and these requests can very quickly become unmanageable during the initial load process.

We recently had a project where we had to import 70,000+ objects into the FIM Portal and the initial load generated 640,000 requests even though we limited the MPR's and workflows that applied during initial loading. This firstly makes it hard to find a specific request, and also slows the performance of the web components down when performing a full search of requests (even causing time out issues in some cases).

In order to limit this from occurring we generally reduce the time that FIM Portal keeps completed request in the Portal before archiving them. The default retention period that requests are retained is set to 30 days. This means that if you generate 200,000 request this week, 700,000 the next week you would have to live with an additional 900,000 requests until the request reach an age of 30 days. You can see how this can become a problem during initial loading as it would:

1. Slow down portal request searches for this time
2. Put an additional storage load on you SQL database ( we had to extend the storage to over 2TB at one client)
3. Put a strain of the index file ( we had to rebuild the index file numerous times)

We usually reduce the retention period to 2-5 days during initial loading, as this gives us the ability to still deal with requests in a easy manner, but have the flexibility to to get rid excessive completed requests quickly.
This should be done prior to starting the initial load, as doing the change after the fact does not change the retention age on completed requests, but only those generated after the change. It should also be noted that these requests are archived and not deleted, but fall outside the scope of search-able requests in the FIM Portal, and can be retrieved via SQL queries or reporting tools.

A well planned initial load can save you many headaches and will contribute to a successful deployment.
For more information on FIM best practises navigate to the following link:

http://technet.microsoft.com/en-us/library/ff608274(WS.10).aspx