HMC CreateFolder

Today I had the chance to get back to some code for CreateFolder which creates a public folder within our Exchange 2007 environment support by Microsoft’s Hosted Messaging and Collaboration frame work. I’ve been highly critical of Microsoft’s code before, and today has taken that to a new level.

Here is what I’m dealing with. First, let’s go to the documentation on TechNet:

Pretty clear that we have a few required parameters and a few optional ones to set some quotas. These optional parameters are what will soon drive me to uncontrollable rage.

Next, let’s look at the example directly from the web service. Here’s what it states we can pass:

POST /mpsws/ManagedEmail2007/Service.asmx HTTP/1.1
Host: prov01.bacon.lightedge.com
Content-Type: text/xml; charset=utf-8
Content-Length: length
SOAPAction: "http://provisioning.microsoft.com/webservice/managedemail2007/CreateFolder"

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <CreateFolder xmlns="http://provisioning.microsoft.com/webservice/managedemail2007">
      <CreateFolderRequest xmlns="http://provisioning.microsoft.com/managedemail2007">
        <Data>
          <organization>string</organization>
          <preferredDomainController>string</preferredDomainController>
          <name>string</name>
          <path>string</path>
          <server>string</server>
          <maxItemSize>string</maxItemSize>
          <postStorageQuota>string</postStorageQuota>
          <storageQuota>string</storageQuota>
        </Data>
      </CreateFolderRequest>
      <sendCredentials>boolean</sendCredentials>
    </CreateFolder>
  </soap:Body>
</soap:Envelope>

Again, everything looks great. postStorageQuota and storageQuota are both there and seem perfectly reasonable.

So let’s actually SEND that command to the web service. Here’s the error I get back:

The element 'CreateFolder_Request' in namespace 'http://provisioning.microsoft.com/exchange2007provider' has invalid child
element 'postStorageQuota' in namespace 'http://provisioning.microsoft.com/exchange2007provider'. List of possible elements expected: 'prohibitPostQuota, server, issueWarningQuota' in namespace 'http://provisioning.microsoft.com/exchange2007provider'.The element 'CreateFolder_Request' has invalid child element 'postStorageQuota' . List of possible elements expected: 'prohibitPostQuota, server, issueWarningQuota' .

You get this is you change PostStorageQuota to prohibitPostQuota.

The element 'CreateFolder_Request' in namespace 'http://provisioning.microsoft.com/exchange2007provider' has invalid child element 'storageQuota' in namespace 'http://provisioning.microsoft.com/exchange2007provider'. List of possible elements expected: 'prohibitPostQuota, server, issueWarningQuota' in namespace 'http://provisioning.microsoft.com/exchange2007provider'.The element 'CreateFolder_Request' has invalid child element 'storageQuota' . List of possible elements expected: 'prohibitPostQuota, server, issueWarningQuota' .

In case you missed it, they changed the variable with in the back end server code, and failed to tell anyone that they made this change. Awesome!

This appeared only after we made the jump up to HMC 4.5 rollup 9. Before the PostStorageQuota and StorageQuota worked as expected. Needless to say, I’m less than impressed that they changed properties on me and failed to update the documentation OR the web service example. I guess figuring this out with trial by fire is the way that Microsoft expects you to learn. Thanks Microsoft!

HMC Resource Management

Today I’m going to talk about the wonderful thing that is the Resource Manger for HMC (Hosted Messaging and Collaboration), the wondering framework from Microsoft for provisioning users for Exchange, OCS and SharePoint automagically.

Now, we tend to do things a bit differently at my current job. For example, I’m pretty sure I’m the only developer in the US that is hitting HMC (again, a Microsoft product) with PHP hosted on our linux based customer portal. That’s right, the Unix guys is the lead developer hitting a very Microsoft-centric product. Normally this would be done by a .Net developer.

So what does the Resource Manager give you? Well, the resource manager allows you to add, edit, remove mail databases for use on the system in the Exchange world which is what we are mainly going to focus on. When you first setup these resources, you set certain parameters for the mail stores. Do you want the domains on there to be tightPack or minStoreCount. TightPack attempts to use the smallest number of mail stores. MinStoreCount attempts to minimize the number of stores allocated to each org. We have gone with the minStoreCount model as we want to have all the users of the same domain live on the same store if possible.

Which is good in theory, but Microsoft has an extremely poor implementation of this.

Here’s an example. There is a domain on our system. The users used to live in 2 mail stores. We moved the users, through HMC calls, to all live in the same mail store. Now, when we create a new user, we do not specify the mail store. We rely on HMC and the resource manager to put the user in the correct location. Unfortunately, Microsoft drops the ball big time here. Instead of putting the user on the correct mail store with the other 30 accounts, it instead picks the old store that has no users on it. It fails to figure out that the mail store has been abandoned by that domain and continues to put users there.

Nice right?

So how do we get around this? Simple, write our own and tell them exactly where to put it. Yes, that’s right, recreating the wheel. Well all know we don’t need a rounder wheel. But in this case, we’re actually improving the functionality of the system.

I can’t go into the detail of how we actually implemented our system. But I can tell you, its already paying off by putting the users in the correct location.

But but but…is that it? Is that all you really solved? Do you really care if the users are spread out? Well, yeah, we do. We want them to be in the same mail store to help save on space for those messages that have the same attachment that went out to every users in the organization. Its not going to save you a ton of space, but it does save a lot more than you think it does.

But the placing of users isn’t the only improvement that we got from writing our own resource manager. One lacking feature of the resource manager is the ability to mark mailstores as non provisionable. I’m sure I’m going to get someone from Microsoft telling me that yes you can, there is a bit you can flip. And I’m here to tell you, they’re right, but they completely missed the boat on actual implementation.

It is true, you can mark a mail database as non provisionable. However, when you create a new user, HMC will ask which databases the domain lives on and picks one of them. note, it doesn’t query for all the databases that are provisionable, it asks for ALL of them for the domain. There is a chance, a very GOOD chance that HMC may automatically pick one of the stores that you have marked as not provisoinable. Awesome I know.

So how do you get around this little gem. Well, you can mess with the resource manager as we ran for a while. We had a script that would run daily and check a list of databases that we had marked to not have new users on them. Then it would check out the free space on the system and mess with the resource manager basically tricking the resource manager that it would be out of space. This had the same effect of not having new users allocated to it on the system as the resource manager would at least check to see if there was enough space on the database when putting a user on there. But even this had its drawbacks. For example, what if you have a system setup where you have plans in place where a user can have a 1GB, 2GB and 4GB mailbox. They have the 1GB plan and want to upgrade. Guess what, if the datastore doesn’t think that there is any more space on the drive, you you CANNOT upgrade them. Why in the world would you ever want someone to upgrade their account and spend more money with you. Crazy talk I know.

I’m sure you’re thinking, how in the heck did you ever come up with these crazy ways of mail database retirement. These suggestions came from Microsoft themselves. Even our premier tech support guy (who is awesome btw) didn’t know that these changes would cause such ass ache. Its like they never ran into a situation where someone would want to mark a database to not have any more new users allocated to it.

Our solution, write our own resource manager. not for the weak of heart, but I’m also not a pansy when it comes to this sort of thing. What we have done is put in place a system that keeps track of where each user, domain and mail store is located, how many users per store and where each domain lives. We can easily query this for the best place to put a user, but also the best place to put a user even if you have retired all the stores that the domain lives on, the system is smart enough to place the user on a new datastore for that particular domain. now when a new user is created, our provisioning system queries the resource manager, gets the appropriate store and we put that once “optional” parameter in HMC in the request XML and basically bypass the pile of shit resource manager that Microsoft has given us.

I’m sure that the big boys out there in this space will laugh at this post. I’m sure that they are way past where we are at. Or they have a bigger team that ran into this problem and put in their own solution. But if you happen to be starting out and are relying on the HMC resource manager, you might want to think again on that one. Eventually, you’ll write your own.

A device attached to the system is not functioning

As you’ve read before, I’ve had a variety of battles with the Hosted Messaging and Collaboration framework from Microsoft.

Today was another day for battle. And an interesting battle it was.

So here is the situation that I was running into. Within my code in the customer portal, I have a notification that is sent out that has a full back trace of what happened on the system, what was inputted and what was the error message that was returned. I of coarse try to give the user a friendly version of the error message to the screen before sending off this plethora of valuable detail. Out of this pile of data, I find the following error message has been returned:

“A device attached to the system is not functioning”

Really? Let’s look at that again. Because surely no one in their right mind would return such a generic error!

“A device attached to the system is not functioning”

Hmm, my clue-by-four hasn’t been brought out recently and plane tickets to Redmond aren’t that expensive right now. I guess I could go out there and start handing out clues. But why would I need to. This error message is clear and concise. A device attached to the system is not functioning. What more could I possibly need from this error message. It gives me everything I want! There’s a device, its one of 30 or so devices in the system, that is not functioning.

HOW ABOUT WHICH DEVICE YOU FUCKTARDS!!!!!!!

So we check out the domain controllers, the MPS servers, the web servers, OCS pools, exchange queues, look at SCOM alerts. Nothing pops out at us. Then I take a look a bit further at the error message. here is the full error message that I get back:

Server was unable to process request. ---> Unable to create 'cn=user@domain.com'./A device attached to the system is not functioning. (Exception from HRESULT: 0x8007001F)/Create Object -- ><response><errorContext description="Unable to create'cn=user@domain.com'./Adevice attached to the system is not functioning. (Exception from HRESULT: 0x8007001F)/Create Object" code="0x8007001f" executeSeqNo="58"><errorSourcenamespace="Error Provider" procedure="SetError"/><errorSourcenamespace="Managed_Active_Directory" procedure="RethrowError_"/><errorSourcenamespace="Managed_Active_Directory"procedure="TryCreateSamObject_"/><errorSourcenamespace="Managed_Active_Directory"
procedure="CreateSamObject_"/><errorSourcenamespace="Managed_Active_Directory" procedure="CreateUser_"/><errorSourcenamespace="Managed_Active_Directory" procedure="CreateUser"/><errorSourcenamespace="Hosted Active Directory"procedure="CreateUser"/></errorContext></response>

The CreateSamObject_ raised a red flag to two. If you search Google long enough you’ll eventually find something from MSDN stating the following:

From MSDN:

User account names are limited to 20 characters and group names are limited to 256 characters. In addition, account names cannot be terminated by a period and they cannot include commas or any of the following printable characters: “, /, \, [, ], :, |, <, >, +, =, ;, ?, *. Names also cannot include characters in the range 1-31, which are nonprintable

Eureka! We have found the issue!!!

The sAMAccountName for our system is pretty generic. Cut down the upn to 16 characters. Add a period and a unique key that we keep for each of our domain names. This way, the chances of you duplicating names both inside and outside of your OU is pretty limited. However, in this case there was an error with retrieving the unique 3 letter key for the end of the name so the sAMAccountName that we were attempting to set looked something like this:

&lt;property name='sAMAccountName' &gt;
        &lt;value&gt;user_domain.&lt;/value&gt;
&lt;/property&gt

This was an easy fix once we found the issue. But man, could there be a crappier error message from HMC? I mean seriously, how do you get from unable to create the sAMAccountName to a device attached to the system is not functioning?!?!?

Anyway, I figured that I would type this up, let the search engines find it and hopefully someone else can benefit from our digging through the depths of HMC.

Ugly Code

With my recent project, I had to do some research for the bits that make up the msRTCSIP-OptionFlags field in Active Directory for OCS users. There were certain operations that are not 100% supported by HMC so often times you have to fill in the gaps. The definition of this field is as follows:

This attribute specifies the different options that are enabled for the user or contact object. This attribute is a bit-mask value of type integer. Each option is represented by a bit. This attribute is marked for Global Catalog replication.

source

Unfortunately, in my searching I found not only the technet article, but I also found this

Here a sample of some of the code that you’ll find….

if strOptionFlag = "" then
objUser.Put "msRTCSIP-OptionFlags", "16"

else

‘If user not enabled then do not enable OCS
if strOptionFlag = "0" then
objUser.Put "msRTCSIP-OptionFlags", "0"
‘If user is enabled for Public IM then keep Public IM enabled and enable RCC
elseif strOptionFlag = "1" then
objUser.Put "msRTCSIP-OptionFlags", "257"
‘If user is enabled for RCC then keep RCC enabled and enable Enhanced Presence
elseif strOptionFlag = "16" then
objUser.Put "msRTCSIP-OptionFlags", "272"
‘If user is enabled for RCC and Public IM then keep RCC & PIM enabled and enable Enhanced Presence
elseif strOptionFlag = "17" then
objUser.Put "msRTCSIP-OptionFlags", "273"
‘If user is already enabled for RCC and Enhanced Presence then keep settings
elseif strOptionFlag = "272" then
objUser.Put "msRTCSIP-OptionFlags", "272"
‘If user is already enabled for RCC, PIM and Enhanced Presence then keep settings
elseif strOptionFlag = "273" then
objUser.Put "msRTCSIP-OptionFlags", "273"
end if
end if

All I have to say is wow! You’ve got to be freaking kidding me. Are there programmers out there that don’t understand what a bitfield is supposed to be or how to do bitwise operations? That’s like the first freaking class in a Computer Science degree. This is where you learn about Big 0, Binary numbers, loops, variables. If you are a programmer reading this blog and you don’t know what a bit field is and how to us it. Google it, read up, and rid the world of ugly ass code like what I found above.

Now technically what this person is doing would “work” since it is setting the integer with an appropriate number. But wow is this the WRONG way to do it. I’m afraid to see this persons code for when they need to check and see if the RCC bit is set for the user. Technically, they should be able to do a simple bitwise operation and check it something along the lines of:

if(($msRTCSIP-OptionFlags & 0x0010) != 0)
{
// RCC is set
}

I have a feeling that this person is doing for the following:

if(value == 273)
{
X is set
Y is set
Z is set
}
else if(value == 257)
{
Y is set
Z is set
}

This is a very simplistic psuedo-code example. But wow…I’m still in shock that this type of code is out there…and posted somewhat recently in October of 2008!

I really hope I run into this guy some day so I can beat him with the clue-by-4.

The need for speed

As I’ve written before, the main engine that we are using to provision users on our Exchange system is Microsoft‘s Hosted Messaging and Collaboration (HMC) framework.

I’ve had a few issues with it and various hoops that I have had to jump through. Many of which I have not had a chance to document on this blog. Those will hopefully come out in the coming months.

Today, I was dealing with a speed issue that we have been having in our customer portal that hits HMC. The page is pretty basic, it gives a listing of all the users, the name, login address, what package they are assigned and then some actions that can be taken on the user such as edit, delete, disable. This is all in house code that shows this information with calls being made to HMC to get the data.

Unfortunately, I made a mistake in my initial development. Originally, I got a listing of all the users and had 90% of the information that I needed. Another call was needed to get the assigned package for each user. If you have 5-10 users, this isn’t a huge deal. Try it with 100, or 1000. It gets really ugly really fast.

On one particular customer that I was looking at, they have 120 users. Their page was loading in 41 seconds. 41 freaking seconds!!! 7 seconds for the AD search to grab the users with mailboxes. Then somewhere between .1 and .3 seconds per user to equal 34 seconds of the page load.

The solution, remove the package display code.

What? But doesn’t the user want to see this data?!?! Maybe. But seeing as they have to go to the edit page anyway to change the package, it really isn’t needed on the account listing page. So why display it there. It slows down the user experience which is more of a negative than having it on the page.

It wasn’t an easy decision to remove this information. I thought of a whole bunch of ways where I could tuck that data away so I could grab it more efficiently. But in the end, less is less. Removing that data cleaned up the look of the page and vastly improved the load times. Its only been in the wild for a for hours so we’ll see what the users have to say about the change.