Recently we had a pretty funny conversation with our Microsoft premier support tech. It was clear that a memo had gone out from up above and he was doing his part to push bing out to everyone that he could. Unfortunately, he forgot he was talking to engineers. We’re smart ass bastards and he should know better.

The conversation went something like this.

Tech: Have you guys tried Bing yet?

Us: Yeah, but I don’t see us switching anytime soon. Google is still the better search engine.

Tech: Really guys? I haven’t found that to be the case. Bing has been really great!

Us: Ok, let’s do an experiment shall we, Go to bing and search for “Exchange 2007 SP2 rollup 1 download”. You’ll find a bunch of blogs talking about the download, but not the actual download page itself. Now go to google, type in the same thing and hit the “I’m feeling lucky” button.

At the time of this conversation, the “I’m feeling lucky” link took you straight to the download page, the bing results did NOT have the download page listed on their first page of results.

Tech: Oh, yeah…I guess we have some things to work out.

Us: We’ll try again once Microsoft figures out how to index their own site.

Unfortunately, that conversation happened a couple of months ago. The results are getting a little bitter for bing, but google is still the king of search.

HMC Resource Management

Today I’m going to talk about the wonderful thing that is the Resource Manger for HMC (Hosted Messaging and Collaboration), the wondering framework from Microsoft for provisioning users for Exchange, OCS and SharePoint automagically.

Now, we tend to do things a bit differently at my current job. For example, I’m pretty sure I’m the only developer in the US that is hitting HMC (again, a Microsoft product) with PHP hosted on our linux based customer portal. That’s right, the Unix guys is the lead developer hitting a very Microsoft-centric product. Normally this would be done by a .Net developer.

So what does the Resource Manager give you? Well, the resource manager allows you to add, edit, remove mail databases for use on the system in the Exchange world which is what we are mainly going to focus on. When you first setup these resources, you set certain parameters for the mail stores. Do you want the domains on there to be tightPack or minStoreCount. TightPack attempts to use the smallest number of mail stores. MinStoreCount attempts to minimize the number of stores allocated to each org. We have gone with the minStoreCount model as we want to have all the users of the same domain live on the same store if possible.

Which is good in theory, but Microsoft has an extremely poor implementation of this.

Here’s an example. There is a domain on our system. The users used to live in 2 mail stores. We moved the users, through HMC calls, to all live in the same mail store. Now, when we create a new user, we do not specify the mail store. We rely on HMC and the resource manager to put the user in the correct location. Unfortunately, Microsoft drops the ball big time here. Instead of putting the user on the correct mail store with the other 30 accounts, it instead picks the old store that has no users on it. It fails to figure out that the mail store has been abandoned by that domain and continues to put users there.

Nice right?

So how do we get around this? Simple, write our own and tell them exactly where to put it. Yes, that’s right, recreating the wheel. Well all know we don’t need a rounder wheel. But in this case, we’re actually improving the functionality of the system.

I can’t go into the detail of how we actually implemented our system. But I can tell you, its already paying off by putting the users in the correct location.

But but but…is that it? Is that all you really solved? Do you really care if the users are spread out? Well, yeah, we do. We want them to be in the same mail store to help save on space for those messages that have the same attachment that went out to every users in the organization. Its not going to save you a ton of space, but it does save a lot more than you think it does.

But the placing of users isn’t the only improvement that we got from writing our own resource manager. One lacking feature of the resource manager is the ability to mark mailstores as non provisionable. I’m sure I’m going to get someone from Microsoft telling me that yes you can, there is a bit you can flip. And I’m here to tell you, they’re right, but they completely missed the boat on actual implementation.

It is true, you can mark a mail database as non provisionable. However, when you create a new user, HMC will ask which databases the domain lives on and picks one of them. note, it doesn’t query for all the databases that are provisionable, it asks for ALL of them for the domain. There is a chance, a very GOOD chance that HMC may automatically pick one of the stores that you have marked as not provisoinable. Awesome I know.

So how do you get around this little gem. Well, you can mess with the resource manager as we ran for a while. We had a script that would run daily and check a list of databases that we had marked to not have new users on them. Then it would check out the free space on the system and mess with the resource manager basically tricking the resource manager that it would be out of space. This had the same effect of not having new users allocated to it on the system as the resource manager would at least check to see if there was enough space on the database when putting a user on there. But even this had its drawbacks. For example, what if you have a system setup where you have plans in place where a user can have a 1GB, 2GB and 4GB mailbox. They have the 1GB plan and want to upgrade. Guess what, if the datastore doesn’t think that there is any more space on the drive, you you CANNOT upgrade them. Why in the world would you ever want someone to upgrade their account and spend more money with you. Crazy talk I know.

I’m sure you’re thinking, how in the heck did you ever come up with these crazy ways of mail database retirement. These suggestions came from Microsoft themselves. Even our premier tech support guy (who is awesome btw) didn’t know that these changes would cause such ass ache. Its like they never ran into a situation where someone would want to mark a database to not have any more new users allocated to it.

Our solution, write our own resource manager. not for the weak of heart, but I’m also not a pansy when it comes to this sort of thing. What we have done is put in place a system that keeps track of where each user, domain and mail store is located, how many users per store and where each domain lives. We can easily query this for the best place to put a user, but also the best place to put a user even if you have retired all the stores that the domain lives on, the system is smart enough to place the user on a new datastore for that particular domain. now when a new user is created, our provisioning system queries the resource manager, gets the appropriate store and we put that once “optional” parameter in HMC in the request XML and basically bypass the pile of shit resource manager that Microsoft has given us.

I’m sure that the big boys out there in this space will laugh at this post. I’m sure that they are way past where we are at. Or they have a bigger team that ran into this problem and put in their own solution. But if you happen to be starting out and are relying on the HMC resource manager, you might want to think again on that one. Eventually, you’ll write your own.

Windows 7 Failure?

So I was checking out CNN today and I found this article titled Mac share grew after Windows 7 debut.

In the article, you’ll see some gems such as this:

If Microsoft (MSFT) was hoping that the launch of Windows 7 would halt the erosion of its operating system market share — and curb further inroads by Apple (AAPL) — there is no evidence that it’s working yet.

In fact, preliminary data released overnight Sunday by Net Applications show Mac OS X’s Internet share growing by 2.73% in October, from 5.12% to 5.26%.

Wow…really? Up 2.73%! Holy crap they’re kicking the crap out of Microsoft. Sell your M$ stock…they’re royally f*cked now!

Not quite.

Microsoft is still at 92.54% of the market share if you can trust these numbers, which are probably pretty accurate. So for those keeping track at home, that means that Microsoft is only up by say 87 points. Its still not even close people. So even though this is a CNN article that links to their site, I wouldn’t read too much into this one. Is Apple making some gains, sure. But they’re still so far behind its not even funny.

And I’m an Apple fan!

One Character == World of Suck

Ladies and Gentlemen, today you are going to learn a lesson on why you do NOT edit the active directory directly for exchange attributes.

A long time ago, we had a very crappy provisioning system for our hosted Exchange 2003 platform. It worked ok, but missed a lot of things that we wanted to have set. They also were kind of pricks when it came to licensing so making a ton of money on the platform was hard to do. So, we decided to roll our own. It wasn’t that hard to reverse engineer what was being set for users, groups and contacts. There were a few obstacles of coarse but we were able to get a pretty good provisioning system setup. However, this too had its faults. Sure we had total control over the code and could update things as we needed. But we were still working in a void. We really didn’t know _everything_ that was happening on the system that needed to actually happen. Plain and simple, we were missing things. Not to mention future services would require the same amount of dev time reverse engineering what needed to be set. That’s not a scalable solution.

So, when it came time to roll out our Exchange 2007 environment, we have moved over to HMC and for the most part, things have been a lot happier.

The Issue
Since we have this shiny new 2007 platform, we thought it was in our best interest to start migrating our users from the 2003 platform and away from the old clunky provisioning. So far we have moved some smaller customers and everything has gone pretty smooth. Certain customers have required the migration to happen in stages so we have come up with a decent solution to make this as smooth as possible. We define the list of users to migrate, setup a temporary domain on the 2003 platform, add smtp aliases to the users on the 2007 platform and then enable forwarding on the 2003 platform to the 2007 temporary smtp alias. All mail still flows as it did before, its just that some users happen to be on the 2007 platform getting their email. Easy as can be right?

Well, the users that we have moved so far have been fairly small customers. So the need for automating certain aspects of the move have been put off. There is a lot of automation that happens, but some of the tasks were put off such as the setting of the forwarding address.

That is, until last week. Last week we were starting to move one of the bigger customers so I setup a script that would take the users that were being moved and automatically set the forwarding address. Here is the code that I had in place to figure out the forwarding address:

            // need to add forwarding address here.
            list($tuname, $tdmn) = explode("@", $user['mail']);
            // new address
            $fwd = "{$tuname}@{$newdomain}\n";

Notice the $fwd variable. This is where the trouble begins. You can ignore the {} as those simply tell the parser that these are valid variables contained within the string.

No, the real issue is at the end of the line. Specifically, the \n. Technically, you can set the AD object to whatever you want. It will take this as a value. However, Exchange does not like this. With forwarding in exchange, you set the forwarding to an object. Typically another user, group of contact. Since we allow for users to forward to multiple people, we create a group automatically and populate that with one or more users, groups or contacts.

Now for the really bad part. I’m not sure if this is just Exchange 2003 or if others are affected as well. But when you forward to a group that contains a single contact object that has invalid characters, your email message with be lost in the ether. Gone, Kaput, Not even an NDR will be generated. We were unable to even find logs that the message came in it failed that spectacularly.

So the moral of the story is, don’t do what we did. Microsoft will tell you the exact same thing. Its dumb to do what we did. You should not be messing with the AD attributes directly in this instance. There is a certain level of error checking that must take place at the upper levels which were missed by both my script, and the provisioning system. But in the same breath, I can’t say for sure that HMC was mature enough to use when we first rolled out the 2003 platform. So whether it is you editing AD directly or someone else doing it, its just as bad. It just depends on how much control you want over the gun pointed at your head.

Business as usual

Over the past couple of years, I have been able to tolerate Microsoft a bit more than I used to. When your primary income relies on people purchasing Exchange and OCS accounts that you provide the back end provisioning and automation for, you quickly realize where your bread is buttered.

But this sort of crap really needs to stop. Yes, its their operating system. But that doesn’t excuse installing add-ons to 3rd party applications and disabling the uninstall options. I’m with the writer of this article, this is a great way to get your customers to not trust you and precisely the reason I haven’t had windows on my desktop for 8 years.

A routine security update for a Microsoft Windows component installed on tens of millions of computers has quietly installed an extra add-on for an untold number of users surfing the Web with Mozilla’s Firefox Web browser.

Earlier this year, Microsoft shipped a bundle of updates known as a “service pack” for a programming platform called the Microsoft .NET Framework, which Microsoft and plenty of third-party developers use to run a variety of interactive programs on Windows.

The service pack for the .NET Framework, like other updates, was pushed out to users through the Windows Update Web site. A number of readers had never heard of this platform before Windows Update started offering the service pack for it, and many of you wanted to know whether it was okay to go ahead and install this thing. Having earlier checked to see whether the service pack had caused any widespread problems or interfered with third-party programs — and not finding any that warranted waving readers away from this update — I told readers not to worry and to go ahead and install it.