Category Archives: Windows

Backup Exec tapes refuse to catalog

I recently wasted a fair amount of time when I needed to restore some files which were older than the 60 days of catalog retention on the Backup Exec media server. Trying a catalog job would fail with a zero byte count after 15 minutes with the error:

e00084c8 – The backup storage device has failed.

Catalog
An unknown error occurred on device “QUANTUM 1”.
V-79-57344-33992 – The backup storage device has failed.

I suspected bad media, so ordered an older media set from storage, but this exhibited the same problem. The error was strange because it implied that the media or the hardware was at fault, but media errors are normally clearly signposted, and often the autoloader front panel will also display this kind of error. I involved Dell Support just to rule this out, but there were no errors with the loader (nor media nor drive).

All the information on this error code I could find on the web was either for genuine hardware faults, much earlier product versions, or irrelevant.

Symantec Support did figure out a solution though – it seemed as if, despite the catalogs no longer being held on the media server for these tapes, there was some kind of corruption in the Backup Exec database.

The workaround was was to select the problem tapes in the Media tab, associate them with the Retired Media Media Set, then right-click and delete them from the media view. This removes any mention of that media from Backup Exec’s database. Then rescan the autoloader’s slots in the Devices tab, and run an Inventory job on them all. Finally, run the Catalog job once again – then it works.

I’m writing this up here, because I have a vague memory of being burned by this issue once before, and it only ever strikes when you’re dealing with a data loss scenario – not a time when you want to be starting up multiple support requests and waiting hours!

Exchange 2007/2010 coexistence in a single server deployment

I’m making a notes here for myself, mainly because a lot of the upgrade/coexistence material is over-complex when you’re dealing with a single server deployment of Exchange. I decided that the lowest impact method to transition to Exchange 2010/2007 coexistence was to leave the public DNS/smarthost settings/MX records unchanged, and simply switch the IP addresses of the servers. So the alias mail.domain.com moves from Exchange Server 2007 to 2010.

  • Choose a public IP address for legacy.domain.com and create a host record in your public DNS.
  • On your border firewall, add this new host to the same rules you have defined for mail.domain.com. Pay attention to any other rules involving this host (permitting traffic from other firewall security zones, like wifi for instance).
  • Make sure your SSL certificate has the following FQDNs (assuming domain.com is your public DNS domain): internal server name (exch2010.domain.local), external server name (mail.domain.com), autodiscover for your main email domains (autodiscover.domain.com), legacy for your main email domain (legacy.domain.com).
  • If any of these are missing, re-key your certificate (GoDaddy lets you change Subject Alternate Names without buying a new cert, as long as you bought the multiple-SAN type). Prepare the cert request using the Exchange 2010 Management Console. When GoDaddy re-key a cert they use the SANs from the original cert request, regardless of whatever is defined in the new CSR. So before you upload the CSR, edit the SANs in the GoDaddy control panel.
  • Import the issued certificate using the Exchange 2010 Management Console.
  • Double-click on the cert you downloaded from GoDaddy. In the Details pane you can double check the Subject Alterative Names, and also the Thumbprint. Make a note of that.
  • Use the Exchange 2010 Management Shell to assign this cert to the main services, replacing the default self-signed one:
    Get-ExchangeCertificate
    Have a look at the current assignments, then:
    Enable-ExchangeCertificate -Thumbprint {Thumbprint} -Services "IIS, IMAP, POP, SMTP"
  • Now we need to import this same cert onto the Exchange 2007 server since it will be known as legacy.domain.com. Importing using the Exchange 2007 Management Shell cmdlet is not sufficient since the private key will not be imported. The only way I know of doing this is to export the cert from your Exchange 2010 server as a .pfx file, making sure to select the option to export the private key. Then on the Exchange 2007 server, create a dummy website with a custom host header so you don’t disrupt the running websites, enable SSL on a random port and import that .pfx certificate. Once that is done you can delete the dummy website.
  • Now in the Exchange 2007 Management Shell run:
    Get-ExchangeCertificate
    to list your certs. Your new cert’s thumbprint should be listed. Now we just need to enable the new cert for the same services the old one was using. e.g.:
    Enable-ExchangeCertificate -Thumbprint {Thumbprint} -Services "IIS, IMAP, POP, SMTP"
  • You could at this point tidy up and use Remove-ExchangeCertificate to remove the old cert but there’s not much point since the Exchange 2007 server will soon be decommissioned.
  • Configure your Exchange 2010 CAS and Hub Transport roles with the external hostname set to mail.domain.com. Do not build a send connector yet (Organization Configuration > Hub Transport).
  • In the Exchange 2007 Management Console, select Server Configuration > Client Access. For each of the service connection points (OWA, Exchange ActiveSync, OAB, etc.) ensure that the URLs are changed from mail.domain.com to legacy.domain.com.
  • There is another service connection point that’s not listed here in the Management GUI – the Exchange Web Services (EWS) virtual directory. Fail to migrate this one and you’ll get this error message from Outlook clients that try to edit their Out of Office settings:

    Your Out of Office settings cannot be displayed, because the server is currently unavailable. Try again later.

  • This one can be updated using the Exchange 2010 Management Shell:
    Get-WebServicesVirtualDirectory -Server EXCH2007 | fl *url
    Get-WebServicesVirtualDirectory -Server EXCH2007 | Set-WebServicesVirtualDirectory -ExternalUrl https://legacy.domain.com/ews/exchange.asmx
  • In the Exchange 2007 Management Console disable all of the Send Connectors (Organization Configuration > Hub Transport), and Receive Connectors (Server Configuration > Hub Transport).
  • Shut down all the Exchange services on both servers, and set the IP address of the Exchange 2007 server to the one you created in DNS for legacy.domain.com. Then reboot.
  • Change the IP address of the Exchange 2010 server to the one you were using for Exchange 2007. Reboot it.
  • Do not manually delete any DNS records at this point!
  • The external smarthost is configured to accept mail only from one of the Exchange servers – mail.domain.com. Rather than edit this, and wait for propagation etc. it’s probably easier just to have your Exchange 2010 server send and receive all external email.
  • To do this, either edit your Send Connector and change the server name, or leave it disabled and create a new connector copying the settings, but with Exchange 2010 as the source server.
  • At this point I had a problem with inbound mail delivery. The queue viewer showed that it was stuck at the Exchange 2010 server, and was failing to resolve internal DNS for the Exchange 2007 server. I had manually deleted the DNS entries for the two Exchange servers when I redefined their IPs. The deletion then synced to the AD/DNS at another business site and the tombstone replicated back later on – deleting the newly updated records.
  • As is fairly typical, the Exchange 2007 server had a separate SMTP relay receive connector defined in Server Configuration > Hub Transport, to allow web servers and monitoring scripts to send email externally. Re-creating this on the Exchange 2010 server proved problematic. It turned out that once this connector was created, the mail flow from Exchange 2007 to 2010 would also bind to it, rather than the DEFAULT connector. Since the Externally Secured option prevents some of the other auth methods being enabled, one of the Exchange 2007 server’s transport queues was stuck with this error:

    451 4.4.0 Primary target IP address responded with: “451 5.7.3 Cannot achieve Exchange Server authentication.” Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or delivery failed to all alternate hosts.

  • The solution was to create a new receive connector on the Exchange 2010 server, specific only to the legacy.domain.com IP address which matched the settings of the DEFAULT one. This can later be deleted when coexistence was is longer needed. Apparently the connectors are bound to in the order of the most specific first.
  • Following this migration work, it transpired that older iPhones on iOS 3.x, and Android handsets are not able to understand Exchange 2007/2010 coexistence and need the mail server name manually updating to legacy.domain.com. Also, Outlook 2003 (there’s always a legacy client with a home user somewhere!) needs to have encryption enabled for server communication.

Unified Windows PE 3.1 builder script for WAIK, with wifi and EFI support

WinPE-with-WLAN

It’s useful to have wifi support with WPA/WPA2 in Windows PE for occasions where you may want to perform an OS install or a salvage on a machine without ethernet built-in: for instance a MacBook Air running Windows or a Sony Vaio P when you don’t have the breakout dongle to hand.

When I started researching wifi support I found that the people who had clearly got it working seemed to be unwilling to share their work. One tool for the job, Holger’s PE Network Manager did not work for me at all, and judging from the dates of the files it had been created for older versions of Windows PE than the current one. There are many third party Windows PE ‘builder’ apps out there that completely replace the official Microsoft WAIK tools. However these seem more aimed at hobbyists than pros and tend to over complicate by adding features that a deployment pro does not really need. Since they’re not-so-trustworthy compiled binaries I don’t feel comfortable using them for work purposes. I decided to carefully work out for myself which registry entries and which additional files are required. There are surprisingly few.

The following unified script will use the standard WAIK tools to create both x86 and x64 builds, or either one individually, and is designed to be run by double-clicking it. Notice the defined variables at the start, particularly the %SOURCE% folder. The script expects the following folders:

  • %SOURCE%\scripts\WinPE – OS build scripts, wifi config XML, optional CA certificates
  • %SOURCE%\drivers\WinPE-x86\CURRENT – drivers
  • %SOURCE%\drivers\WinPE-x64\CURRENT
  • %SOURCE%\tools\WinPE-x86 – optional tools such as GImageX, or apps from portableapps.com
  • %SOURCE%\tools\WinPE-x64
  • %SOURCE%\tools\WinPE-x86\WLAN – the WLAN system files see highlighted section in main script below
  • %SOURCE%\tools\WinPE-x64\WLAN
 

It also requires your WAIK installation to be updated to Windows PE 3.1 using the WAIK supplement for Windows 7 SP1.

I had originally wanted to connect WinPE clients to a WPA-Enterprise network but although Windows PE now supports 802.1x, it appears to work only for wired connections (see my comment on that post).

There is no GUI to manage the WLAN service, but I did find the necessary registry mod to enable the hooks already present in netsh.exe. You will need to use a working Windows 7 PC which is configured for the WLAN of your choice, then export its profile. You cannot connect to a wifi network in Windows PE without having done this. The PSK must not be encrypted since it will be imported onto a different machine (you need to run this with Administrator privileges or else the key parameter will be ignored):

netsh wlan show profiles
netsh wlan export profile name="YOURWLANSSIDHERE" folder="C:\temp" key=clear

Copy this XML file into the folder %SOURCE%\scripts\WinPE in the build tree. When Windows PE boots, you simply type wifi. This wifi.cmd script which is built by the main script will automatically import any XML profiles, list the available wifi networks, and display some example netsh syntax:

netsh wlan connect name=YOURWLANSSIDHERE ssid=YOURWLANSSIDHERE
 

The script

Save as %SOURCE%\scripts\Build_WinPE.cmd

:: Build_WinPE.cmd
::
:: This script will build x86 and x64 Windows PE, automatically
:: collecting drivers from the relevant folders within the
:: unattended installation, building WIM and ISO images, and
:: will also upload the WIM images to the deployment server(s).
::
:: DO NOT cancel this script in progress as you can end up with
:: orphaned locks on files inside mounted WIM images which
:: usually require a reboot of the server to clear.
::

@echo off
setlocal ENABLEDELAYEDEXPANSION

::variables
     set SOURCE=G:\unattended
     set PE_TEMP=C:\temp
     ::WinPE feature pack locale
     set PL=en-US
     ::commma separated list for WDS_SERVERS
     set WDS_SERVERS=FILESERVER1,FILESERVER2
::end variables

if not exist "C:\Program Files\Windows AIK\Tools\PETools\*.*" (
     echo This script requires the Windows Automatic Install Kit to be installed
     echo Download it from http://www.microsoft.com/download/en/details.aspx?id=5753
     echo.
     pause
     goto :eof
)
if "%1"=="relaunch" (
     call :BUILD_WINPE %2 %3 %4
     goto :eof
)
if "%1"=="unmount" (
     :: use this if you have a problem with the script and there are WIMs still mounted
     dism /Unmount-Wim /MountDir:"%PE_TEMP%\WinPE-x86\mount" /discard
     dism /Unmount-Wim /MountDir:"%PE_TEMP%\WinPE-x64\mount" /discard
     goto :eof
)
:prompt
cls
set /P SELECTION=Build WinPE for which CPU architecture (x64, x86, both)? [x64]: 
if "%SELECTION%"=="" set SELECTION=x64
if "%SELECTION%"=="x64" (
     start "Building Windows PE for x64 - NEVER CANCEL THIS SCRIPT IN PROGRESS" cmd /c "%0" relaunch x64 amd64
     goto :eof
)
if "%SELECTION%"=="x86" (
     start "Building Windows PE for x86 - NEVER CANCEL THIS SCRIPT IN PROGRESS" cmd /c "%0" relaunch x86 i386
     goto :eof
)
if "%SELECTION%"=="b" set SELECTION=both
if "%SELECTION%"=="both" (
     ::opening both instances of this script simultaneously seems to cause race conditions with dism.exe
     start /wait "Building Windows PE for x86 - NEVER CANCEL THIS SCRIPT IN PROGRESS" cmd /c "%0" relaunch x86 i386 nopause
     start "Building Windows PE for x64 - NEVER CANCEL THIS SCRIPT IN PROGRESS" cmd /c "%0" relaunch x64 amd64
     goto :eof
)
goto :prompt

:BUILD_WINPE
set PE_ARCH=%1
set PE_ARCH_LONG=%2
echo on
set PATH=%PATH%;C:\Program Files\Windows AIK\Tools\PETools\;C:\Program Files\Windows AIK\Tools\%PROCESSOR_ARCHITECTURE%
set PATH=%PATH%;C:\Program Files\Windows AIK\Tools\Servicing
rd /s /q %PE_TEMP%\WinPE-%PE_ARCH%

::Microsoft haven't used consistent naming in WAIK
if "%PE_ARCH%"=="x86" set WAIK_ARCH=%PE_ARCH%
if "%PE_ARCH%"=="x64" set WAIK_ARCH=%PE_ARCH_LONG%

call copype.cmd %WAIK_ARCH% %PE_TEMP%\WinPE-%PE_ARCH%
::package path
set PP=%ProgramFiles%\Windows AIK\Tools\PETools\%WAIK_ARCH%\WinPE_FPs
::image path
set IP=%PE_TEMP%\WinPE-%PE_ARCH%\mount
echo on
dism /Mount-Wim /WimFile:"%PE_TEMP%\WinPE-%PE_ARCH%\winpe.wim" /Index:1 /MountDir:"%IP%"
dism /image:"%IP%" /Add-Package /PackagePath:"%PP%\winpe-scripting.cab" /PackagePath:"%PP%\%PL%\winpe-scripting_%PL%.cab" /PackagePath:"%PP%\winpe-wmi.cab" /PackagePath:"%PP%\%PL%\winpe-wmi_%PL%.cab" /PackagePath:"%PP%\winpe-mdac.cab" /PackagePath:"%PP%\%PL%\winpe-mdac_%PL%.cab" /PackagePath:"%PP%\WinPE-HTA.cab" /PackagePath:"%PP%\%PL%\WinPE-HTA_%PL%.cab" /PackagePath:"%PP%\WINPE-DOT3SVC.CAB" /PackagePath:"%PP%\%PL%\WINPE-DOT3SVC_%PL%.CAB"
dism /image:"%IP%" /Add-Driver /driver:"%SOURCE%\drivers\WinPE-%PE_ARCH%\CURRENT" /Recurse
copy "%ProgramFiles%\Windows AIK\Tools\PETools\%WAIK_ARCH%\bootsect.exe" "%IP%\Windows"
copy /y "%SOURCE%\scripts\WinPE\*.*" "%IP%\Windows\System32"
copy "%SOURCE%\tools\WinPE-%PE_ARCH%\*.*" "%IP%\Windows\System32"
copy "%ProgramFiles%\Windows AIK\Tools\%WAIK_ARCH%\*.*" "%IP%\Windows\System32"


::add WLAN components
reg load HKLM\PE-BUILD-SYSTEM "%IP%\Windows\System32\config\SYSTEM"
reg load HKLM\PE-BUILD-SOFTWARE "%IP%\Windows\System32\config\SOFTWARE"
set KEY=HKLM\PE-BUILD-SYSTEM\ControlSet001\Enum\Root\LEGACY_WLANSVC
reg add %KEY% /v NextInstance /t REG_DWORD /d 1 /f
reg add %KEY%\0000 /v Service /t REG_SZ /d Wlansvc /f
reg add %KEY%\0000 /v Legacy /t REG_DWORD /d 1 /f
reg add %KEY%\0000 /v ConfigFlags /t REG_DWORD /d 0 /f
reg add %KEY%\0000 /v Class /t REG_SZ /d LegacyDriver /f
reg add %KEY%\0000 /v ClassGUID /t REG_SZ /d {8ECC055D-047F-11D1-A537-0000F8753ED1} /f
reg add %KEY%\0000 /v DeviceDesc /t REG_SZ /d "@%%SystemRoot%%\System32\wlansvc.dll,-257" /f
set KEY=HKLM\PE-BUILD-SYSTEM\ControlSet001\services\wlansvc
reg add %KEY% /v DisplayName /t REG_SZ /d "@%%SystemRoot%%\System32\wlansvc.dll,-257" /f
reg add %KEY% /v ErrorControl /t REG_DWORD /d 1 /f
reg add %KEY% /v Group /t REG_SZ /d TDI /f
reg add %KEY% /v ImagePath /t REG_EXPAND_SZ /d "%%SystemRoot%%\system32\svchost.exe -k LocalSystemNetworkRestricted" /f
reg add %KEY% /v Start /t REG_DWORD /d 2 /f
reg add %KEY% /v Type /t REG_DWORD /d 32 /f
reg add %KEY% /v Description /t REG_SZ /d "@%%SystemRoot%%\System32\wlansvc.dll,-258" /f
reg add %KEY% /v DependOnService /t REG_MULTI_SZ /d "nativewifip\0RpcSs\0Ndisuio\0Eaphost" /f
reg add %KEY% /v ObjectName /t REG_SZ /d LocalSystem /f
reg add %KEY% /v ServiceSidType /t REG_DWORD /d 1 /f
reg add %KEY% /v RequiredPrivileges /t REG_MULTI_SZ /d "SeChangeNotifyPrivilege\0SeImpersonatePrivilege\0SeAuditPrivilege\0SeTcbPrivilege\0SeDebugPrivilege" /f
reg add %KEY% /v FailureActions /t REG_BINARY /d 2c0100000000000000000000030000001400000001000000c0d4010001000000e09304000000000000000000 /f
reg add %KEY%\Enum /v 0 /t REG_SZ /d "Root\LEGACY_WLANSVC\0000" /f
reg add %KEY%\Enum /v Count /t REG_DWORD /d 1 /f
reg add %KEY%\Enum /v NextInstance /t REG_DWORD /d 1 /f
reg add %KEY%\Parameters /v ServiceDll /t REG_EXPAND_SZ /d "%%SystemRoot%%\System32\wlansvc.dll" /f
reg add %KEY%\Parameters /v ServiceDllUnloadOnStop /t REG_DWORD /d 1 /f
reg add %KEY%\Parameters /v ServiceMain /t REG_SZ /d WlanSvcMain /f
reg add HKLM\PE-BUILD-SOFTWARE\Microsoft\NetSh /v wlancfg /t REG_SZ /d wlancfg.dll /f
call :REG_MULTI_SZ-add "HKLM\PE-BUILD-SOFTWARE\Microsoft\Windows NT\CurrentVersion\Svchost" LocalSystemNetworkRestricted wlansvc
xcopy /s "%SOURCE%\tools\WinPE-%PE_ARCH%\WLAN\*.*" "%IP%\Windows"
@echo off
:: These are the files that are required for WLAN (take them from Windows 7 machines of both CPU archs)
:: Put them in %SOURCE%\tools\WinPE-%PE_ARCH%\WLAN and create the same relative folder structure from C:\Windows onwards
:: so the file C:\Windows\inf\netnwifi.inf is copied to %SOURCE%\tools\WinPE-%PE_ARCH%\WLAN\inf\netnwifi.inf
::     C:\Windows\inf\netnwifi.inf
::     C:\Windows\inf\netvwififlt.inf
::     C:\Windows\inf\netvwifimp.inf
::     C:\Windows\l2schemas\wlan_policy_v1.xsd
::     C:\Windows\l2schemas\wlan_profile_v1.xsd
::     C:\Windows\l2schemas\wlanap_profile_v1.xsd
::     C:\Windows\schemas\availablenetwork\availablenetworkinfo.xsd
::     C:\Windows\system32\certutil.exe
::     C:\Windows\system32\wlanapi.dll
::     C:\Windows\system32\wlancfg.dll
::     C:\Windows\system32\wlanhlp.dll
::     C:\Windows\system32\wlanmsm.dll
::     C:\Windows\system32\wlansec.dll
::     C:\Windows\system32\wlansvc.dll
::     C:\Windows\system32\wlanui.dll
::     C:\Windows\system32\wlgpclnt.dll
::     C:\Windows\system32\drivers\nwifi.sys
::     C:\Windows\system32\drivers\vwififlt.sys
::     C:\Windows\system32\drivers\vwifimp.sys
::     C:\Windows\system32\en-US\certutil.exe.mui (or your locale's equivalent)
::     C:\Windows\system32\en-us\wlanapi.dll.mui
::     C:\Windows\system32\en-us\wlancfg.dll.mui
::     C:\Windows\system32\en-us\wlansvc.dll.mui
::     C:\Windows\system32\en-us\wlanui.dll.mui
::     C:\Windows\system32\en-us\wlgpclnt.dll.mui
echo on
reg unload HKLM\PE-BUILD-SYSTEM
reg unload HKLM\PE-BUILD-SOFTWARE
:: build wifi.cmd
(
     echo @echo off
     echo drvload X:\WINDOWS\Inf\netvwifimp.inf
     echo drvload X:\WINDOWS\Inf\netvwififlt.inf
     echo drvload X:\WINDOWS\Inf\netnwifi.inf
     echo netcfg -c s -i ms_nativewifip
     echo echo Importing detected CA certificate^(s^).
     echo for %%%%i in ^(*.cer^) do certutil -addstore root %%%%i
     echo echo.
     echo net start dot3svc
     echo net start wlansvc
     echo for %%%%i in ^("Wireless Network Connection*.xml"^) do netsh wlan add profile filename="%%%%i"
     echo netsh wlan show networks
     echo echo.
     echo echo use "netsh wlan" to manage the wifi connection like so:
     echo echo   netsh wlan connect name=YOURWLANHERE ssid=YOURWLANHERE
     echo echo   ipconfig /renew
     echo echo.
) > "%IP%\Windows\System32\wifi.cmd"

dism /Unmount-Wim /MountDir:"%IP%" /commit
imagex /export /boot /compress fast "%PE_TEMP%\WinPE-%PE_ARCH%\winpe.wim" 1 "%PE_TEMP%\WinPE-%PE_ARCH%\ISO\sources\boot.wim"
@echo off

::Mac OS BootCamp will look for autorun.inf in order to validate this disk as a Windows Installer CD
::adding this allows us to start unattended installs using WinPE
date /T > "%PE_TEMP%\WinPE-%PE_ARCH%\ISO\autorun.inf"

::x64 bootable ISO includes both BIOS & EFI boot loaders
if "%PE_ARCH%" == "x64" (
     set CD_CMD=oscdimg -m -o -u2 -udfver102 -bootdata:2#p0,e,b"%PE_TEMP%\WinPE-%PE_ARCH%\etfsboot.com"#pEF,e,b"%PE_TEMP%\WinPE-%PE_ARCH%\efisys.bin" "%PE_TEMP%\WinPE-%PE_ARCH%\ISO" "%PE_TEMP%\WinPE-%PE_ARCH%\winpe_%PE_ARCH_LONG%.iso"
) else ( 
     set CD_CMD=oscdimg -n -b"%PE_TEMP%\WinPE-%PE_ARCH%\etfsboot.com" "%PE_TEMP%\WinPE-%PE_ARCH%\ISO" "%PE_TEMP%\WinPE-%PE_ARCH%\winpe_%PE_ARCH_LONG%.iso"
)
echo on
%CD_CMD%
@echo off

::rename the WIM file to avoid having multiple image files on the WDS server with the same filename
ren "%PE_TEMP%\WinPE-%PE_ARCH%\ISO\sources\boot.wim" boot_%PE_ARCH_LONG%.wim
del "%PE_TEMP%\WinPE-%PE_ARCH%\winpe.wim"

for %%i in (%WDS_SERVERS%) do (
     echo.
     echo Adding/updating boot image on WDS server: %%i
     :: try to add the image first, if that fails then replace existing
     wdsutil /Verbose /Progress /Add-Image /ImageFile:"%PE_TEMP%\WinPE-%PE_ARCH%\ISO\sources\boot_%PE_ARCH_LONG%.wim" /Server:%%i /ImageType:Boot /Name:"Microsoft Windows PE (%PE_ARCH%)" || wdsutil /Verbose /Progress /Replace-Image /Image:"Microsoft Windows PE (%PE_ARCH%)" /ImageType:Boot /Architecture:%PE_ARCH% /ReplacementImage /Name:"Microsoft Windows PE (%PE_ARCH%)" /ImageFile:"%PE_TEMP%\WinPE-%PE_ARCH%\ISO\sources\boot_%PE_ARCH_LONG%.wim" /Server:%%i
     echo.
)
::rename the WIM back again so bootable USB devices can be created
ren "%PE_TEMP%\WinPE-%PE_ARCH%\ISO\sources\boot_%PE_ARCH_LONG%.wim" boot.wim
echo *******************************************************************
echo WDS boot image(s) updated
echo.
echo A bootable ISO of this image has been created at:
echo   %PE_TEMP%\WinPE-%PE_ARCH%\winpe_%PE_ARCH_LONG%.iso
echo.
echo To create a bootable USB key, use diskpart.exe to create a FAT32 partition
echo and mark it active, then copy the contents of this folder to its root:
echo   %PE_TEMP%\WinPE-%PE_ARCH%\ISO
echo.
echo FAT32 is required for EFI support.
echo.
if "%3"=="nopause" goto :eof
pause
goto :eof

:REG_MULTI_SZ-add
::subroutine to append a value to a multiple string value Registry entry
setlocal ENABLEEXTENSIONS
set KEY=%1
set VALUE=%2
for /f "tokens=2*" %%a in ('reg query %KEY% /v %VALUE% /t REG_MULTI_SZ ^| FIND "REG_MULTI_SZ"') do set DATA=%%b
set DATA=%DATA%\0%3
reg add %KEY% /v %VALUE% /t REG_MULTI_SZ /d %DATA% /f
 
 

Exchange 2010/2007 mixed mode – broken UM

This one was a very obscure issue. I recently installed the first Exchange 2010 server in our infrastructure and haven’t really configured anything on it yet. It turns out that at around the same time our Exchange 2007 Unified Messaging stopped working but it took a while for anyone to notice. Long enough in fact that I strongly suspected our PABX system was at fault, especially since it has recently had some hardware failures.

The symptoms were a total inability to leave messages on the UM server, but strangely it could still be contacted by dialing the voicemail number. The audio message was something like ‘we cannot connect you at this time’. The fact that an Exchange voice prompt can be heard strongly suggests that the telecoms system is ok.

The event log errors on the Exchange server are 1082:

Event Type: Error
Event Source: MSExchange Unified Messaging
Event Category: UMService
Event ID: 1082
Date: 04/10/2011
Time: 16:05:38
User: N/A
Computer: EXCH-01
Description:
The Unified Messaging server was unable to submit messages to a Hub Transport server because there is no Hub Transport server available to process the request with UM header file “C:\Program Files\Microsoft\Exchange Server\UnifiedMessaging\voicemail429a59b-e196-4b25-940d-796c736fb93d.txt”. Make sure that there is a Hub Transport server located in the same Active Directory site as the UM server. In addition, make sure that the Microsoft Exchange Transport service is started on the Hub Transport server.

and 1185:

Event Type: Warning
Event Source: MSExchange Unified Messaging
Event Category: UMCore
Event ID: 1185
Date: 04/10/2011
Time: 16:06:48
User: N/A
Computer: EXCH-01
Description:
The Unified Messaging server was unable to submit a message to Hub Transport server “EXCH-01” because the following error occurred: Unexpected SMTP server response. Expected: 220, actual: 500, whole response: 500 5.3.3 Unrecognized command.

Firstly I tried increasing the Exchange UM log level but that wasn’t helpful at all. Error 500 implies some sort of auth issue so I double-checked all the certificates were valid (Get-ExchangeCertificate | fl). Then I started reading. There isn’t a huge amount on this subject (most of what’s written relates specifically to 2010) but I found this excellent write-up by Terence Luk. His were the same symptoms I was seeing, but even creating a new Hub Transport Receive Connector didn’t fix it.

What was different about my infrastructure? Well in my environment all the roles (including UM) are on the same server. So I had to telnet from the Exchange server itself to test properly. And then I noticed something strange. My Exchange server was referring to itself by the IP of one of its iSCSI NICs, which are on a private IP range. Why? No duplicate A records in DNS from spurious interface registrations, no Client or Microsoft Networks assigned on the iSCSI NICs. A simple ipconfig seemed to list them in an unusual order, with the actual Local Area Connection in the middle. This was probably the cause. Next stop Eric Reitz’s How to set/view the NIC bind order in Windows. A new menu I never new existed until this point!

And sure enough, this fixed it. The Exchange 2007 server’s NIC binding order had been like this for well over a year and never caused any issues, but I think that the addition of the first Exchange 2010 server must have raised the security, probably implementing TLS for inter-role UM traffic where it wasn’t used before.

Disabling Windows IPv6 tunnelling

Just after doing an Active Directory migration to Windows 2008 R2 native mode I started experiencing odd DNS registration behaviour. The DNS for this domain is hosted in AD, so the migration meant that all DNS servers were now 2008 R2 (replacing 2003 R2). As a result, some 2008 R2 member servers decided to register IPv6 addresses for their automatic 6to4 tunnel interfaces in DNS which led to connectivity problems for clients (which are all Windows 7, and therefore IPv6 aware). What was really strange is that this was not consistent. Some servers were unaffected, despite being in the same subnet.

The workaround was to disable all IPv6 tunnelling over IPv4. Here are the netsh commands since they vary enough to be confusing…

 

to disable:

netsh interface ipv6 6to4 set state state=disabled
netsh interface ipv6 set teredo type=disabled
netsh interface ipv6 isatap set state state=disabled
 

to query:

netsh interface ipv6 6to4 show state
netsh interface ipv6 show teredo state
netsh interface ipv6 isatap show state
 

Cisco wifi WPA2-Enterprise PEAP authentication with Active Directory

Business users increasingly expect full LAN access while working wirelessly around the workplace. On account of the perceived weakness of WPA cryptography many network administrators will tend to offer a separate guest network over wifi, but not the full corporate LAN. I was one of those people until I got this working six months ago. This solution supports Mac, PC clients, together with iOS devices (iPhone, iPad), and I would guess Linux too since it’s based on open standards.

There is a pretty comprehensive Cisco configuration example document on this subject, but there isn’t much information on the web apart from that. There are several drawbacks to that guide:-

  • it only outlines the GUI config of a centralised Cisco Wireless LAN Controller (not standalone Cisco wifi access points)
  • it deals with Windows Server 2003 IAS rather than its more current replacement: NPS, as introduced by Windows Server 2008.
  • it assumes that the client will be running Cisco wifi client software (presumably it pre-dates the introduction of Windows’ own Wireless Network control panel introduced with XP Service Pack 1).

The access points authenticate using PEAP which is EAP inside a TLS session, therefore a working PKI is required for creating the NPS server’s certificate. If you don’t already have it, install the Active Directory Certificate Services role to one of your Windows domain servers. As I mentioned in my LDAPS guide, that whole process is somewhat outside the scope of this blog post but do heed Microsoft’s warning:

Warning Before you install a certification authority (CA), you should be aware that you are creating or extending a public key infrastructure (PKI). Be sure to design a PKI that is appropriate for your organization. See PKI Design Brief Overview for additional information.

 

Windows PKI Problems

DCs should auto-enroll for their own certificates once that’s up and running. However I had huge difficulties with this. I installed the Active Directory Certificate Services role on a Windows Server 2008 R2 Domain Controller. I soon discovered that none of the other 2008 R2 DCs could auto-enroll for certificates (the Event Log reported RPC server unavailable in the failure event which was quite misleading). I could not manually enroll for certificates using the Certificates MMC snap-in either.

This held me up for a long time, and I was able to find several sources stating that the solution was to add the Domain Controllers to the group CERTSVC_DCOM_ACCESS, which didn’t work for me. In the end I found the solution in this Technet answer by Joson Zhou. Somebody in the distant past had apparently tampered the Active Directory group membership of Builtin\Users (CN=Users,CN=Builtin,DC=domain,DC=com) and had removed Interactive and Authenticated Users. It had been like this for years with no adverse impact. Since a DC has no local accounts it will use this AD group as its definition of which users have User privileges. I suppose that up until this time no service that would have been affected had been installed on a DC.
I called a business contact to verify against an external Active Directory and, sure enough, those missing groups should indeed have been members. The moment I corrected this I was able to enroll for a DC certificate. Sadly the Technet thread is locked so I wasn’t able to thank Joson, and unfortunately it seems you can’t send private messages. I don’t understand why anyone would be inclined to interfere with default Active Directory groups – searching only reveals this post. The guy asks if it’s best practice to remove Authenticated Users from that group and is pretty roundly slapped down. Perhaps people did this with NT 4.0, which my domain dates from.

I also had a recurring issue in the Event Logs where these DCs would fail to pull down the Certificate Revocation Lists from Microsoft and fail to refresh the Trusted Root certificates. Microsoft KB931125 was the only thing that fixed that. However, while troubleshooting this problem I ended up installing the Trusted Root CA update pack (from here I think), assuming I’d be able to uninstall it later if it didn’t improve things. Unfortunately it cannot be uninstalled, and it caused me another huge hold-up later on.

 

Configuring Network Policy Server

Once your Domain Controller has a certificate you can install the Network Policy Server role. Each access point (or only your WLC if you use LWAPP access points) will need to be on a static IP address, and have an entry in the RADIUS clients section of the NPS management MMC. When generating Pre-Shared Keys note that it seems Cisco devices do not tolerate keys containing a dollar sign, at least not in their WebUI.

NPS RADIUS client settings

The behaviour for Windows computers is that before logon the PC will authenticate and connect using its computer account (providing connectivity to the domain), then when the user logs in the wifi network is re-connected in that user context. You will therefore probably need an NPS Network Policy item for both cases (NPS -> Policies -> Network Policies) which will allow you to apply different settings to each (timeouts, etc.):

NPS policy for wireless computers
NPS policy for wireless users

In each of the policies be sure to select PEAP as the only EAP authentication type, with EAP-MSCHAP v2 as below, selecting your server’s certificate in the drop-down.
NPS server PEAP encryption settings

Make sure that your Connection Request Policies are not preventing connections (NPS -> Policies -> Connection Request Policies) – I think they are disabled by default.

 

Access Point Configuration

I won’t bother to describe the Cisco WLC configuration, since that is basically identical to Cisco’s own guide. However, one problem I had in testing was that I removed and re-added the RADIUS server settings on the WLC and I had forgotten that you need to specify your NPS server twice – once for Authentication, and separately once again for Accounting (assuming you’re using accounting), which is not the case in the Server Manager GUI on standalone access points:

Wireless LAN Controller RADIUS server settings

Not everyone has the budget for a WLC, or you might, like me, have combination of a WLC at one site but not at others. It’s not practical to use a single WLC for multiple sites since all LWAPP traffic is trunked back to the controller – you wouldn’t want that load on your site-to-site links. Here is a sample multiple SSID configuration for a standalone Cisco 1131AG Access Point (AIR-AP1131AG-E-K9). I’m not going to provide Cisco access point GUI screenshots, but if you want them the first part of the following guide covers that well:
http://blog.laurence.id.au/2010/03/running-peap-with-cisco-aeronet-1231g.html

My example access point config below relates to the following schematic. If you’re going to apply this config to a bare unit, substitute the IPs and VLANs to match your environment and remove the lines with keys or passwords.

Sample access point config schematic

!
! Last configuration change at 17:13:17 UTC Tue Feb 22 2011 by admin
! NVRAM config last updated at 17:14:03 UTC Tue Feb 22 2011 by admin
!
version 12.4
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname test-ap1
!
logging rate-limit console 9
enable secret 5 3242576EABCD3F3254$2
!
aaa new-model
!
!
aaa group server radius rad_eap
 server 172.16.2.50 auth-port 1645 acct-port 1646
!
aaa group server radius rad_mac
!
aaa group server radius rad_acct
 server 172.16.2.50 auth-port 1645 acct-port 1646
!
aaa group server radius rad_admin
!
aaa group server tacacs+ tac_admin
!
aaa group server radius rad_pmip
!
aaa group server radius dummy
!
aaa authentication login eap_methods group rad_eap
aaa authentication login mac_methods local
aaa authorization exec default local
aaa accounting network acct_methods start-stop group rad_acct
!
aaa session-id common
ip domain name domain.com
ip name-server 208.67.222.222
ip name-server 208.67.220.220
!
!
dot11 syslog
dot11 vlan-name Wifi-Corp-WLAN vlan 20
dot11 vlan-name Wifi-Guest-WLAN vlan 6
!
dot11 ssid Corp-WLAN
   vlan 20
   authentication open eap eap_methods
   authentication key-management wpa version 2
   accounting acct_methods
   mbssid guest-mode
!
dot11 ssid Guest-WLAN
   vlan 6
   authentication open
   authentication key-management wpa
   guest-mode
   mbssid guest-mode
   wpa-psk ascii 7 123475652142BCD122
!
power inline negotiation prestandard source
!
!
username admin privilege 15 secret 5 131412ABDEFEE42323
!
!
bridge irb
!
!
interface Dot11Radio0
 no ip address
 no ip route-cache
 !
 encryption mode ciphers tkip
 !
 encryption vlan 20 mode ciphers aes-ccm
 !
 encryption vlan 6 mode ciphers aes-ccm tkip
 !
 ssid Corp-WLAN
 !
 ssid Guest-WLAN
 !
 mbssid
 station-role root
!
interface Dot11Radio0.20
 encapsulation dot1Q 20 native
 no ip route-cache
 bridge-group 1
 bridge-group 1 subscriber-loop-control
 bridge-group 1 block-unknown-source
 no bridge-group 1 source-learning
 no bridge-group 1 unicast-flooding
 bridge-group 1 spanning-disabled
!
interface Dot11Radio0.6
 encapsulation dot1Q 6
 no ip route-cache
 bridge-group 6
 bridge-group 6 subscriber-loop-control
 bridge-group 6 block-unknown-source
 no bridge-group 6 source-learning
 no bridge-group 6 unicast-flooding
 bridge-group 6 spanning-disabled
!
interface Dot11Radio1
 no ip address
 no ip route-cache
 !
 encryption mode ciphers tkip
 !
 encryption vlan 20 mode ciphers aes-ccm
 !
 encryption vlan 6 mode ciphers aes-ccm tkip
 !
 ssid Corp-WLAN
 !
 ssid Guest-WLAN
 !
 no dfs band block
 mbssid
 channel dfs
 station-role root
!
interface Dot11Radio1.20
 encapsulation dot1Q 20 native
 no ip route-cache
 bridge-group 1
 bridge-group 1 subscriber-loop-control
 bridge-group 1 block-unknown-source
 no bridge-group 1 source-learning
 no bridge-group 1 unicast-flooding
 bridge-group 1 spanning-disabled
!
interface Dot11Radio1.6
 encapsulation dot1Q 6
 no ip route-cache
 bridge-group 6
 bridge-group 6 subscriber-loop-control
 bridge-group 6 block-unknown-source
 no bridge-group 6 source-learning
 no bridge-group 6 unicast-flooding
 bridge-group 6 spanning-disabled
!
interface FastEthernet0
 no ip address
 no ip route-cache
 duplex auto
 speed auto
!
interface FastEthernet0.20
 encapsulation dot1Q 20 native
 no ip route-cache
 bridge-group 1
 no bridge-group 1 source-learning
 bridge-group 1 spanning-disabled
!
interface FastEthernet0.6
 encapsulation dot1Q 6
 no ip route-cache
 bridge-group 6
 no bridge-group 6 source-learning
 bridge-group 6 spanning-disabled
!
interface BVI1
 ip address 172.16.2.238 255.255.255.0
 no ip route-cache
!
ip default-gateway 172.16.2.254
ip http server
no ip http secure-server
ip http help-path http://www.cisco.com/warp/public/779/smbiz/prodconfig/help/eag
ip radius source-interface BVI1
snmp-server community public RO
radius-server attribute 32 include-in-access-req format %h
radius-server host 172.16.2.50 auth-port 1645 acct-port 1646 key 7 14154A12D56B57231FDC235331292321631B212212034332
radius-server vsa send accounting
bridge 1 route ip
!
!
!
line con 0
line vty 0 4
!
sntp server 130.159.196.118
sntp server 195.66.148.150
sntp server 84.45.87.84
sntp broadcast client
end
 
 

The Final PKI Hurdle

Each time I tried to authenticate I got an Schannel Event Log error on the NPS server of “The message received was unexpected or badly formatted.” which exactly matched the symptoms described in Microsoft KB933430, although that was only intended for Windows Server 2003. This was confusing, but according to that article:

When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificate authorities trusted for client authentication and remove those that do not really need to be trusted.

I had been working on this issue sporadically and I had only got as far as getting the Enterprise CA online before going on holiday. Only later did I make the connection and remember the updated Trusted Root CA pack that I had loaded on in desperation. Consulting the Certificates MMC snap-in I discovered that the server had 304 trusted root CAs instead of nine! Windows Server 2008 and 2008 R2 do have a more generous storage allowance for sending CA certificates in the PEAP handshake but clearly 304 certificates was too much. Using another server as a reference machine I manually deleted all the superfluous CA certificates and I could finally authenticate via wifi!

 

Non-Domain Client Configuration

When connecting from non-domain machines or iOS devices, the issuing CA for the NPS server’s certificate will not be trusted. In iOS you will be prompted to accept it manually, but the situation is more complicated in Windows 7 (I haven’t tested older OS versions). In my tests the connection would be rejected upon providing the additional user credentials. You will need to export the CA server certificate by running the following command on your CA:

certutil -ca.cert c:\temp\domain-SERVERNAME-CA.cer
 

On the Windows 7 client use the Certificates MMC snap-in to import it into the user’s Trusted Root CA Certificate Store (run MMC.EXE, File -> Add/Remove Snap-in -> Certificates -> My User Account -> Trusted Root Certification Authorities -> Certificates -> right-click -> All Tasks -> Import).

Now if you try to connect you will get the following message:

The Credentials provided by the server could not be validated. We recommend that you terminate the connection and contact your administrator with the information provided in the details. You may still connect but doing so exposes you to a security risk by a possible rogue server.

 

I found that I couldn’t connect even if I ignored this warning. To get around this, you will need to create the wireless network manually using the following settings:

Manual connection settings for Windows 7 (part 1)

Make sure to manually select your CA which should be listed as below:

Manual connection settings for Windows 7 (part 2)

 

Windows AD Domain Client Configuration by Group Policy

Domain workstations can have their wireless networking configs entirely managed by Group Policy, including network preference order, auto connect options, whether to cache credentials and more. Domain members also implicitly trust the Domain CA. Be aware that Windows XP workstations are configured by a separate policy to Windows Vista and Windows 7. Both Group Policies are located at Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Wireless Network (IEEE 802.11) Policies. It’s all pretty self-explanatory, and the security and encryption settings are broadly identical to those in the screenshots above, except that you do not need to specify the CA.

Remember to add your designated wifi users and computers to the AD groups you created for the NPS Network Policies!

iPad, iPhone, and Mac OS X L2TP/IPsec VPN to Windows Server 2008 R2

I spent quite a while experimenting with L2TP over IPsec with my iPad 2, and surprisingly found no useful guides as to how to configure it. Judging by what I could find online, most people simply give up and use PPTP instead which has significant security vulnerabilities. Here’s a concise comparison of PPTP versus L2TP/IPsec which describes that weakness:
http://www.ivpn.net/pptp-vs-l2tp-vs-openvpn.php

I had considered using Apple’s support for Cisco IPsec but that would have meant exposing the core switch where I work. It’s old enough to make that a bad idea. The Juniper Netscreen firewall only supports L2TP with certificates and not Pre-Shared Key so that was also ruled out. This post will outline how to configure Windows Server 2008 R2’s NPS/RRAS role to host L2TP/IPsec connections which will allow iPads and iPhones to connect securely into your Windows infrastructure without the need for additional client software.

Firstly, it’s likely that your NPS/RRAS server is behind a perimeter firewall. If this is the case you’ll need to grant IPsec traffic access from the public internet. Using details from this Technet post I created the following custom service object on the Netscreen firewall, and allowed it inbound to the RRAS server (IP protocols 50 and 51, UDP 500 and 4500). For initial testing though you should probably create a rule to allow all traffic to and from your test client.

IPSec service definition

I am going to assume a knowledge of both NPS and RRAS. For more information on those, other guides exist. As far as I have been able to discover, it seems that the iPad only supports Pre-Shared Key authentication for the IPsec tunnel, rather than certificates-based. The VPN connection settings GUI in Mac OS 10.6 for instance will allow either method, but not in iOS. It may be possible to force your way around this with the iPhone Configuration Utility (designed for applying corporate settings to iOS) but information is pretty scant. I did find a long forum thread about certificate auto-enrollment, and a Microsoft Directory Services team blog post, but I suspect they may relate more to 802.1x:
https://discussions.apple.com/message/10402090
http://blogs.technet.com/b/askds/archive/2010/11/22/ipad-iphone-certificate-issuance.aspx

The L2TP/IPsec Pre-Shared Key is configured by right-clicking on the top level of Routing and Remote Access in Server Manager -> Properties -> Security tab:
Pre-Shared Key for IPsec tunnel

It’s useful to keep your VPN clients on a different subnet to your servers, however multihoming with several NICs can cause problems, particularly if your RRAS server is also a Domain Controller. You can define a subnet for this purpose in the IPv4 tab here, but you will need to remember to add a static route entry on your router pointing traffic for this subnet to the RRAS server.

RRAS client subnet settings

In Server Manager -> NPS -> Policies -> Network Policies create a policy with the following settings, making sure to set the encryption settings. As this Microsoft KB article makes clear, these options actually ensure that IPsec gets used, with the different grades here representing different algorithm proposal combinations. The iPad supports the maximum encryption setting.

NPS Policy Settings for L2TP/IPSec

Lastly, the Mac OS X and iOS VPN client configuration is pretty self-explanatory. Make sure to use the Pre-Shared Key that you defined on the RRAS server (referred to here as Secret):

iPad L2TP VPN configuration

I would at this point like to thoroughly recommend iTap RDP as being the best iOS Remote Desktop client I have seen. It has NLA authentication support, a universal iPad/iPhone binary, and by far the most intuitive controls which really puts it ahead of the competition.

UPDATE – I was hoping to use this VPN configuration for all clients, but it seems that Mac OS clients cannot connect. Mac OS apparently didn’t use the standard L2TP UDP port 1701. Someone compiled a fix for Snow Leopard but I could not get it to work. It’s possible that this is all out of date information though.

UPDATE 2 – I did some more troubleshooting from home and discovered that when a tunnel is initiated from a second device on my home network while another tunnel is already up, all further connection attempts then fail for a long while, even when the RRAS server is rebooted. This would suggest that the Netscreen firewall at my work still considers the original session open, and thus it will eventually timeout after 30 minutes. This behaviour had disrupted my Mac OS X test results. Using verbose logging on the Mac and looking at the NPS log I could see that Mac OS X 10.6.8 VPN client does not accept the 128bit encryption setting. Permitting 56bit encryption allows Macs to connect, but perhaps older versions of Mac OS could have difficulties. I have updated the policy settings screenshot above.

UPDATE 3 – I realised that although NATed clients could connect, clients with public addresses could not. I have amended the destination ports for IP protocols 50 and 51 in the firewall IPsec definition screenshot (it had defaulted to 0-0 rather than 0-65535 for some reason). I have verified that this VPN works for Windows XP clients, Windows 7, Mac OS X 10.6, and Mac OS X 10.5, as well as iPhones (mine’s on iOS 3.1.3) and iPads. Once connected to the RRAS server you cannot interact with that server directly, so make sure that the RRAS server’s own DNS settings do not refer to itself as a primary (assuming it’s also a DNS server) – these DNS entries will be inherited by all VPN clients.

Secure Active Directory authentication for non-domain DMZ web sites using LDAPS

Being able to have people log into websites using credentials they already use is a huge advantage. Hosting those websites in DMZ zones away from domain controllers means that you need to do this via LDAP, but this isn’t secure since the query results are sent in plain text. Enter LDAPS, which from the research I did online seems to be the most misunderstood technology I have used to date, with very few documented examples.

It’s a pretty simple idea – wrap LDAP in a TLS session. The LDAP client connects to the server. The server sends over its certificate, which is used to authenticate it as being the actual server the client expects. This is critical since the webserver is delegating authentication to it. A TLS session is established, and the client binds to the directory with credentials sent in plain text, but TLS provides the security against eavesdropping.

What is absolutely baffling is how many big companies are completely oblivious to getting this working correctly. I deal with one of the big players in the email scanning space which offers LDAPS authentication so customers can use their normal Active Directory credentials to check their spam. I had bought a £500 VeriSign certificate expressly for this purpose, but when this certificate expired I discovered that this company still had connectivity to the domain controller. They are clearly not even checking the chain of trust on the certificate, even though they allow it to authenticate users! It would be relatively trivial to exploit this to impersonate a particular domain controller, insert an appropriate dummy user account, and hack any of their customers who use federated authentication. Security by DNS is not security (think man-in-the-middle, or cache-poisoning).

I have also worked with a supposedly major website CMS system which cannot get this right, despite claims by the vendor that it supports LDAPS. It categorically doesn’t – LDAP on a different TCP port is still LDAP.

Concept summary

There seems to be a lot of conflicting information out there on how this works and how the certificates are involved, so this is the overview as I understand it:

  • The LDAPS client initiates a connection to the DC on TCP636 specifiying the DC’s FQDN (IP address or hostname only are not sufficient).
  • The DC checks through its Local Computer certificate store for a certificate with the same FQDN, then signs using its private key as proof of ID. The certificate is sent to the client, so the client gets the DC’s public key.
  • The LDAPS client checks the chain of trust on that certificate. If the CA is not a trusted public one (like VeriSign) the client system will need to have the issuing CA’s own certificate pre-loaded into its Local Computer Trusted Root Certification Authorities certificate store. Remember, the web server is not in the Active Directory domain so it won’t implicity trust that CA.
  • Once the client trusts that the DC is genuine, the SSL session is established and the client can bind to the directory using plain text Active Directory account credentials.
 

Configuring your Domain Controller for LDAPS

This knowledgebase article outlines the steps the DC goes through to select a certificate, and includes details of how to create the CSR for external certification authorites: http://support.microsoft.com/kb/321051

The information there is pretty good, and there aren’t many gotchas. If you’re aiming for third party servers to bind to your AD then you will probably want an SSL certificate from a commercial CA. Be aware that you will need a certificate with a Subject Alternate Name (to have two hostnames registered) if you use a different internal FQDN for that server. The cheap public CAs like GoDaddy could not offer this the last time I enquired (admittedly about two years ago). If the website that will be binding to your directory is your own then you won’t want to buy a certificate. If you don’t already have it, install the Active Directory Certificate Services role to one of your Windows domain servers. That whole process is somewhat outside the scope of this blog post. Heed Microsoft’s warning from their Technet LDAP over SSL wiki:

Warning Before you install a certification authority (CA), you should be aware that you are creating or extending a public key infrastructure (PKI). Be sure to design a PKI that is appropriate for your organization. See PKI Design Brief Overview for additional information.

DCs should auto-enroll for their own certificates once that’s up and running. As per that Microsoft KB article that’s all you need to do. DCs will respond to LDAPS requests providing they can find a valid SSL certificate. I did run into some issues setting up my PKI which I have documented in another post.

If you’re using your own Active Directory domain’s CA, export the CA’s own certificate by running:

certutil -ca.cert c:\temp\domain-SERVERNAME-CA.cer
 

Configuring the DMZ webserver

Assuming your domain controller isn’t in public DNS, add its FQDN to the hosts file on your DMZ webserver.

Use the Certificates MMC snap-in to import the CA’s certificate you exported earlier (run MMC.EXE, File -> Add/Remove Snap-in -> Certificates -> Computer Account -> Local Computer -> Trusted Root Certification Authorities -> Certificates -> right-click -> All Tasks -> Import).

Example C# code snippet for the website:

NetworkCredential networkCredentials = new System.Net.NetworkCredential(adminUsername, adminPassword);

LdapDirectoryIdentifier ldapDirectoryIdentifier = new LdapDirectoryIdentifier(ldapServer, ldapPort);
LdapConnection ldapConnection = new LdapConnection(ldapDirectoryIdentifier, networkCredentials, AuthType.Basic);

ldapConnection.SessionOptions.SecureSocketLayer = true;
ldapConnection.Bind();
 

Verifying your LDAPS server using OpenLDAP

Since in my case I was not developing the .Net code myself, I found it useful to use OpenLDAP to demonstrate to our developers that the server and firewall were definitely working as expected. I had been unable to get the normal Windows LDAP troubleshooting tool LDP.EXE to work and it’s not particularly verbose during failures (as it turns out, I hadn’t been using the FQDN). OpenLDAP is extremely easy to install on any Linux distro with a package management system. I ended up using my Synology NAS at home. The format of the issuing CA certificate needs to be slightly different though (base64 encoded). On the CA server, use:

certutil -ca.cert c:\temp\domain-SERVERNAME-CA.cer
certutil -encode c:\temp\domain-SERVERNAME-CA.cer c:\temp\domain-SERVERNAME-CA.pem
 

Add the following line to OpenLDAP’s ldap.conf (mine was /opt/etc/openldap/ldap.conf):

TLS_CACERT /opt/etc/openldap/domain-SERVERNAME-CA.pem
 

Make sure of course to copy the .pem file to that location.
The ldapsearch syntax is quite a minefield but my sample query was as follows:

ldapsearch -x -v -H 'ldaps://ldap.domain.com' -b 'dc=domain,dc=com' -s base -D 'user@domain.com' -W
 

which binds to the directory as user@domain.com (you are prompted for a password), and requests only the top level directory objects.

It seems to be very difficult to compile OpenLDAP for Windows, in fact it’s actually impossible using MinGW (one of the dependencies CyrusSASL needs Visual Studio to compile).

I tried to use this binary build of OpenLDAP for Windows but it ignores ldap.conf because the people who compiled it forgot to define the variable sysconfdir in the configure script. I was able to use the excellent Process Monitor to track a failed filesystem read to a path featuring the static entry $SYSCONFDIR$ as ldapsearch.exe is invoked. Unfortunately as such you can’t add the CA cert location to the config.

Batch script for recursive FFmpeg transcoding

I recently had to shrink around 50GB of MP3 audio recordings that were sitting in a nested folder structure on a web server. Having experimented to find more appropriate LAME encoder settings for spoken word content I needed to transcode the files whilst keeping the existing ID3 tags intact. FFmpeg can do this nicely using libmp3lame, whereas LAME by itself cannot. Armed with my own compile of FFmpeg, I created a drag & drop batch script to recursively work through a folder structure transcoding MP3 and WAV files and writing out the resulting MP3 files in the same folder structure with an amended top level folder name. It will accept multiple files or folders being dragged and dropped. You could adapt this script to whatever task you’re using FFmpeg for.

::MP3 transcoding to more sensible default quality settings for spoken word
::32kHz mono VBR @ approx. 64Kbps
::
::pcloadletter.co.uk

@echo off
setlocal ENABLEDELAYEDEXPANSION
::parse multiple command lines (so multiple targets can be dragged and dropped at once)
:commandlineloop
if "%~1"=="" goto :continue
call :mainloop %1
shift
goto :commandlineloop
:continue
echo.
pause
goto :eof

::main loop we run for each command line arg of the script
:mainloop
set folderpath=
if "%~1"=="" (
  echo Drag and drop files or folders onto this script.
  goto :eof
)
set folderpath=%~1
echo "%1" | find /I ".mp3" && (
  goto :fileonly
)
echo "%1" | find /I ".wav" && (
  goto :fileonly
)

::folders
echo.
echo.
echo Processing folder %1
echo ________________________________________________________________________________
echo.

::we need to find the last folder level in the folderpath
::batch scripting is horrible so we need to use a hacky way to get the last token
::(from http://stackoverflow.com/questions/5473840/last-token-in-batch-variable)
set temp_string=%~1
set count=0

::iterate parsing the tokens and trimming the string until it's empty, while counting how many loops
:loopcounter
for /F "tokens=1* delims=\" %%a in ( "%temp_string%" ) do (
  set /A count+=1
  set temp_string=%%b
  goto loopcounter
)
for /F "tokens=%count% delims=\" %%i in ( "%~1" ) do set folder=%%i

::append the top level folder name with "-optimized" so we create a new folder tree
for /R "%folderpath%" %%i in (*.mp3 *.wav) do (
  set sourcepath=%%~di%%~pi
  set destpath=!sourcepath:%folder%=%folder%-optimized!
  md "!destpath!"
  ffmpeg -i "%%~fi" -acodec libmp3lame -aq 8 -ar 32000 -ac 1 "!destpath!%%~ni.mp3"
)
goto :eof

::individual files
:fileonly
for %%i in ("%folderpath%") do (
  ffmpeg -i "%folderpath%" -acodec libmp3lame -aq 8 -ar 32000 -ac 1 "%%~di%%~pi%%~ni-optimized.mp3"
)
 

There are several neat little tricks here. Multiple folders can be dragged onto the script because it uses SHIFT to work through each command line argument. The next hurdle is to get the last part of the folder name(s) being dragged. Though this would be simple in any Unix-like shell, but to do this in batch without relying on any additional tools proved to be quite tricky. Though tokens can be parsed by FOR, finding the last one (which we need) requires us to count how many tokens there are. Only then can the last one be selected. The folder name string substitution is done by the SET command (also used in my file renaming script). Delayed variable expansion means that the variables between the the exclamation marks are evaluated once per loop rather than the default of once per script execution (more info here). I hadn’t realised until today that you can create a folder structure several layers deep with a single MD command. This avoids having to iterate through all subfolders – we can use FOR’s /R switch to handle the recursion in a single line. For more information on some of the variables containing tildes like %%~pi and %%~ni, try running FOR /?. The script also first runs the FFmpeg subroutine with ECHO in front of the commands so you can double-check the syntax before proceeding.

There are of course much more efficient ways of compressing spoken word content than MP3 these days – AAC-HEv2 for instance, but that will rule out all but the latest audio playback devices.
The source MP3 recordings were in 44.1kHz mono @ 128Kbps. This was definitely overkill for speech. Just by opting for 64Kbps I could halve that. However I couldn’t drop the sample rate below 32kHz, as this is the lowest legal limit of MPEG-1 layer III. This info is tricky to find but the LAME encoder reveals it in its extended help:

MPEG-1   layer III sample frequencies (kHz):  32  48  44.1
bitrates (kbps): 32 40 48 56 64 80 96 112 128 160 192 224 256 320
MPEG-2   layer III sample frequencies (kHz):  16  24  22.05
bitrates (kbps):  8 16 24 32 40 48 56 64 80 96 112 128 144 160
MPEG-2.5 layer III sample frequencies (kHz):   8  12  11.025
bitrates (kbps):  8 16 24 32 40 48 56 64

 

I think it’s safe to assume that most MP3 players can handle variable bit rate. As far as I can remember only the first generation ones couldnt from early last decade – the sort of ones which only had around 64MB of storage. I don’t think excluding players this old is much of a concern.

Playing around with VBR quality settings in LAME we can see that with -V 9 we get a small file (12.9MB becomes 3.7MB) however it’s actually an MPEG-2 layer III file which as you can see above supports lower sample rates. I’m not certain of the wider compatibility of this sub-type of MP3 file, and besides it does sound considerably worse than the lowest MPEG-1 layer III settings which is –V 8 (12.9MB becomes 5.2MB). This proved to be the best compromise between size and quality.

Compiling x64/x86 FFmpeg on 64bit Windows with libfaac and libmp3lame

Once I had compiled FFmpeg for 32bit Windows, I discovered that it’s probably ten times more difficult to find information about doing the same with free tools on 64bit editions of Windows. A perfect topic for PCLOADLETTER then!

For building FFmpeg we must use the build environment MinGW-w64 which is a separate version of MinGW. The complication is that even when compiling natively you need to use configuration options as if you’re cross compiling (for host type x86_64-w64-mingw32). It’s not strictly necessary with all packages, but some will fail to build without these parameters. As a result you have to search around trying to find all the necessary configure commands for each package, and they’re not always identical. It’s a slow and frustrating process. What’s even more confusing, is that some builds of MinGW-w64 are targetted at 32bit, some 64bit, and some both. What is more, I had to try three different ones before I found one that could compile LAME without errors (though that could be LAME’s fault).

Most people will want a toolchain with the flexibility to create both x64 and x86 binaries, so that is what I shall describe in this guide.

On the MinGW-w64 downloads site browse to the Multilib Toolchains (Targetting Win32 and Win64) section, and download the latest megasoft78 binary build prefixed with mingw-w64-bin_x86_64-mingw. Extract the contents of this archive to the root of your C: drive (which at time of writing will create a folder named mingw64_4.5.2_multilib).

There is no installer – it’s not as automated as the 32bit distribution of MinGW. I spent a long time trying to compile LAME with the official MSYS build and eventually gave up, chosing instead to use a working one made by a third party. Download MSYS_MinGW_GCC_460_x86-x64_Full.7z from xhmikosr.1f0.de. Extract that archive into C:\mingw64_4.5.2_multilib so you have a subfolder in there named MSYS.

Update – I now realise that this archive also includes the whole GCC toolchain as well, but by then I’d already written the guide :)

Download the latest DirectX SDK. Install it, but only select the following component:
DirectX SDK installation options

This document provides useful background information to the uninitiated. Crucially it explains that when you compile with a particular host type defined, most configure scripts will look for a compiler whose filename is prefixed with that host type. So gcc.exe might be present in the toolchain distro as x86_64-w64-mingw32-gcc.exe. The toolchain I just recommended you download does not contain prefixes, but we shall need them since FFmpeg’s configure script expects them. On Windows Vista or later (which should cover most 64bit Windows systems) we can use mklink.exe to create symbolic links. We’ll also copy some missing DirectX headers into the toolchain and configure MSYS.

Start a Command Prompt as Administrator:

cd \mingw64_4.5.2_multilib\bin\
for %i in (*.exe) do mklink x86_64-w64-mingw32-%i %i
copy /y "%DXSDK_DIR%\Include\*.*" C:\mingw64_4.5.2_multilib\x86_64-w64-mingw32\include\
cd \mingw64_4.5.2_multilib\msys\postinstall
pi
 

This will launch the post-install script which will configure a filesystem mount for /mingw. When asked, enter your mingw installation path as c:/mingw64_4.5.2_multilib

Once that’s complete, launch the MSYS shell from your Administrator command prompt:

C:\mingw64_4.5.2_multilib\MSYS\msys.bat
 

Download the source .tar.gz files of:

Copy those downloaded files to C:\mingw64_4.5.2_multilib\msys\home\administrator (since you’re running in the Administrator user context). In the MSYS shell this is your home directory (cd ~).

In the MSYS shell, run:

tar xvfz faac-1.28.tar.gz
tar xvfz lame-3.98.4.tar.gz
tar xvfz ffmpeg-HEAD-6fd00e9.tar.gz
 

For a 64bit compile of FFmpeg with libfaac and libmp3lame:

cd ~/faac-1.28
./configure --prefix=/usr/local/x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --enable-static --disable-shared --with-mp4v2=no
make clean && make
make install
cd ..
cd lame-3.98.4
./configure --prefix=/usr/local/x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --enable-static --disable-shared --disable-decoder --enable-nasm
make clean && make
make install
cd ..
cd ffmpeg-HEAD-6fd00e9
CPPFLAGS="$CPPFLAGS -I/usr/local/x86_64-w64-mingw32/include" ./configure --extra-ldflags='-L/usr/local/x86_64-w64-mingw32/lib' --prefix=/usr/local/x86_64-w64-mingw32 --enable-static --disable-shared --disable-ffplay --disable-ffserver --enable-memalign-hack --enable-libmp3lame --enable-nonfree --enable-libfaac --arch=x86_64 --enable-runtime-cpudetect --enable-w32threads --cross-prefix=x86_64-w64-mingw32- --target-os=mingw32
make clean && make
make install
 

The compiled ffmpeg.exe will be in C:\mingw64_4.5.2_multilib\MSYS\local\x86_64-w64-mingw32\bin.

In summary what’s happening is we’re compiling FAAC without MP4v2 support (compilation halts if that’s enabled), and we’re making sure the binaries are stored in /usr/local/x86_64-w64-mingw32. Likewise for LAME. FFmpeg needs to be told where to find these new libraries (and headers too). We’re also installing it to /usr/local/x86_64-w64-mingw32, so as to keep x64 and x86 binaries separated.

 

For a 32bit compile of FFmpeg with libfaac and libmp3lame:

cd ~/faac-1.28
CFLAGS="-m32" CXXFLAGS="-m32" ./configure --prefix=/usr/local/i686-pc-mingw32 --host=x86_64-w64-mingw32 --enable-static --disable-shared --with-mp4v2=no
make clean && make
make install
cd ..
cd lame-3.98.4
CFLAGS="-m32" CXXFLAGS="-m32" ./configure --prefix=/usr/local/i686-pc-mingw32 --host=x86_64-w64-mingw32 --enable-static --disable-shared --disable-decoder --enable-nasm
make clean && make
make install
cd ..
cd ffmpeg-HEAD-6fd00e9
CPPFLAGS="$CPPFLAGS -I/usr/local/i686-pc-mingw32/include" ./configure --extra-cflags=-m32 --extra-ldflags='-m32 -L/usr/local/i686-pc-mingw32/lib' --prefix=/usr/local/i686-pc-mingw32 --enable-static --disable-shared --disable-ffplay --disable-ffserver --enable-memalign-hack --enable-libmp3lame --enable-nonfree --enable-libfaac --arch=x86_32 --enable-runtime-cpudetect --enable-w32threads --cross-prefix=x86_64-w64-mingw32- --target-os=mingw32
make clean && make
make install
 

The compiled ffmpeg.exe will be in C:\mingw64_4.5.2_multilib\MSYS\local\i686-pc-mingw32\bin.

This is pretty much the same as before, but CFLAGS=”-m32″ and CXXFLAGS=”-m32″ force the 64bit-native C and C++ compilers to produce 32bit binaries. I found that useful option at the bottom of this FFmpeg wiki page. As usual, FFmpeg has its own way of specifying most of these settings (except CPPFLAGS). It is also instructed to use 32bit libraries via –extra-ldflags=’-m32′.

 

Footnote

Until I gave up and used the MSYS distro from xhmikosr.1f0.de, LAME persistently failed to compile with the error undefined reference to `init_xrpow_core_sse’. I tried a newer version of NASM than 2.09.08, replacing nasm.exe with yasm.exe, using LAME 3.98.2 instead of 3.98.4, disabling NASM in the configure, using the Unix makefile, all to no avail. Apart from a problem someone had compiling a Mac universal binary (fixed by using compiler options I could not use with MinGW), the only other link I could find about this is here, and that’s also unsolved. Even taking the nasm.exe (which I notice is 32bit) from the working MSYS bundle and trying it with the SourceForge ‘official’ MSYS bundle, I cannot get LAME to compile. I had no such problem when using the normal MinGW on my 32bit PC from my previous post on this subject, despite compiling my own nasm.exe. If you have any insight into why this might be, please leave a comment!