Exchange 2010/2007 mixed mode – broken UM

This one was a very obscure issue. I recently installed the first Exchange 2010 server in our infrastructure and haven’t really configured anything on it yet. It turns out that at around the same time our Exchange 2007 Unified Messaging stopped working but it took a while for anyone to notice. Long enough in fact that I strongly suspected our PABX system was at fault, especially since it has recently had some hardware failures.

The symptoms were a total inability to leave messages on the UM server, but strangely it could still be contacted by dialing the voicemail number. The audio message was something like ‘we cannot connect you at this time’. The fact that an Exchange voice prompt can be heard strongly suggests that the telecoms system is ok.

The event log errors on the Exchange server are 1082:

Event Type: Error
Event Source: MSExchange Unified Messaging
Event Category: UMService
Event ID: 1082
Date: 04/10/2011
Time: 16:05:38
User: N/A
Computer: EXCH-01
Description:
The Unified Messaging server was unable to submit messages to a Hub Transport server because there is no Hub Transport server available to process the request with UM header file “C:\Program Files\Microsoft\Exchange Server\UnifiedMessaging\voicemail429a59b-e196-4b25-940d-796c736fb93d.txt”. Make sure that there is a Hub Transport server located in the same Active Directory site as the UM server. In addition, make sure that the Microsoft Exchange Transport service is started on the Hub Transport server.

and 1185:

Event Type: Warning
Event Source: MSExchange Unified Messaging
Event Category: UMCore
Event ID: 1185
Date: 04/10/2011
Time: 16:06:48
User: N/A
Computer: EXCH-01
Description:
The Unified Messaging server was unable to submit a message to Hub Transport server “EXCH-01” because the following error occurred: Unexpected SMTP server response. Expected: 220, actual: 500, whole response: 500 5.3.3 Unrecognized command.

Firstly I tried increasing the Exchange UM log level but that wasn’t helpful at all. Error 500 implies some sort of auth issue so I double-checked all the certificates were valid (Get-ExchangeCertificate | fl). Then I started reading. There isn’t a huge amount on this subject (most of what’s written relates specifically to 2010) but I found this excellent write-up by Terence Luk. His were the same symptoms I was seeing, but even creating a new Hub Transport Receive Connector didn’t fix it.

What was different about my infrastructure? Well in my environment all the roles (including UM) are on the same server. So I had to telnet from the Exchange server itself to test properly. And then I noticed something strange. My Exchange server was referring to itself by the IP of one of its iSCSI NICs, which are on a private IP range. Why? No duplicate A records in DNS from spurious interface registrations, no Client or Microsoft Networks assigned on the iSCSI NICs. A simple ipconfig seemed to list them in an unusual order, with the actual Local Area Connection in the middle. This was probably the cause. Next stop Eric Reitz’s How to set/view the NIC bind order in Windows. A new menu I never new existed until this point!

And sure enough, this fixed it. The Exchange 2007 server’s NIC binding order had been like this for well over a year and never caused any issues, but I think that the addition of the first Exchange 2010 server must have raised the security, probably implementing TLS for inter-role UM traffic where it wasn’t used before.

Disabling Windows IPv6 tunnelling

Just after doing an Active Directory migration to Windows 2008 R2 native mode I started experiencing odd DNS registration behaviour. The DNS for this domain is hosted in AD, so the migration meant that all DNS servers were now 2008 R2 (replacing 2003 R2). As a result, some 2008 R2 member servers decided to register IPv6 addresses for their automatic 6to4 tunnel interfaces in DNS which led to connectivity problems for clients (which are all Windows 7, and therefore IPv6 aware). What was really strange is that this was not consistent. Some servers were unaffected, despite being in the same subnet.

The workaround was to disable all IPv6 tunnelling over IPv4. Here are the netsh commands since they vary enough to be confusing…

 

to disable:

netsh interface ipv6 6to4 set state state=disabled
netsh interface ipv6 set teredo type=disabled
netsh interface ipv6 isatap set state state=disabled
 

to query:

netsh interface ipv6 6to4 show state
netsh interface ipv6 show teredo state
netsh interface ipv6 isatap show state
 

Cross-compiling FFmpeg with librtmp for Intel CPU Synology NAS

I used this method to compile the FFmpeg binary for inclusion in my Serviio package, because I don’t have a NAS with an Intel CPU. Several people who tried to help were getting strange errors while trying to compile on their NAS units.

To begin, install 64bit Ubuntu Linux (I used a VM). As a test I first successfully cross-compiled ARM binaries using the same method but using the ARM Synology toolchain, since I was able to test the results. Once I got it working I did the following:

#-----set up Synology toolchain
cd ~/Downloads
wget http://sourceforge.net/projects/dsgpl/files/DSM%203.1%20Tool%20Chains/Intel%20x86%20Linux%202.6.32/gcc420_glibc236_pineview.tgz
tar xvfz gcc420_glibc236_pineview.tgz
sudo mv i686-linux-gnu /usr/local
sudo mv x86_64-linux-gnu /usr/local
#-----I think the Synology toolchain is anticipating being run on i686-linux-gnu, so we need 32bit libraries to even run the binaries
#-----http://maketecheasier.com/run-32-bit-apps-in-64-bit-linux/2009/08/10
sudo apt-get install ia32-libs
export PATH="/usr/local/x86_64-linux-gnu/bin:${PATH}"

#-----libz static lib
wget http://zlib.net/zlib-1.2.5.tar.gz
tar xvfz zlib-1.2.5.tar.gz
cd zlib-1.2.5
#-----http://www.crosscompile.org/static/pages/ZLib.html
#-----http://tinc-vpn.org/examples/cross-compiling-64-bit-windows-binary/
./configure --prefix=/usr/local/x86_64-linux-gnu --static
sed -i 's/^CC=gcc/CC=x86_64-linux-gnu-gcc/g' Makefile
sed -i 's/^LDSHARED=gcc/LDSHARED=x86_64-linux-gnu-gcc/g' Makefile
sed -i 's/^CPP=gcc -E/CPP=x86_64-linux-gnu-gcc -E/g' Makefile
sed -i 's/^AR=ar rc/AR=x86_64-linux-gnu-ar rc/g' Makefile
sed -i 's/^RANLIB=ranlib/RANLIB=x86_64-linux-gnu-ranlib/g' Makefile
make libz.a
cp libz.a /usr/local/x86_64-linux-gnu/lib
cp *.h /usr/local/x86_64-linux-gnu/include/
cd ..

#-----libbz2 static lib
wget http://bzip.org/1.0.6/bzip2-1.0.6.tar.gz
tar xvfz bzip2-1.0.6.tar.gz
cd bzip2-1.0.6
sed -i 's/^CC=gcc/CC=x86_64-linux-gnu-gcc/g' Makefile
sed -i 's/^AR=ar rc/AR=x86_64-linux-gnu-ar rc/g' Makefile
sed -i 's/^RANLIB=ranlib/RANLIB=x86_64-linux-gnu-ranlib/g' Makefile
make
make install PREFIX=/usr/local/x86_64-linux-gnu
cd ..

#-----libmp3lame static lib
wget http://sourceforge.net/projects/lame/files/lame/3.98.4/lame-3.98.4.tar.gz
tar xvfz lame-3.98.4.tar.gz
cd lame-3.98.4
./configure --prefix=/usr/local/x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --build=x86_64-linux-gnu --enable-static --disable-shared --disable-decoder --enable-nasm
make
make install

#-----libssl & libcrypto static libs
ipkg install openssl-dev
wget http://www.openssl.org/source/openssl-0.9.8p.tar.gz
tar xvfz openssl-0.9.8p.tar.gz
cd openssl-0.9.8p
./config --prefix=/usr/local/x86_64-linux-gnu no-shared
sed -i 's/^CC= gcc/CC= x86_64-linux-gnu-gcc/g' Makefile
sed -i 's/^AR= ar /AR= x86_64-linux-gnu-ar /g' Makefile
sed -i 's/^ARD=ar /ARD=x86_64-linux-gnu-ar /g' Makefile
sed -i 's/^RANLIB= \/usr\/bin\/ranlib/RANLIB=x86_64-linux-gnu-ranlib/g' Makefile
sed -i 's/^MAKEDEPPROG= gcc/MAKEDEPPROG= x86_64-linux-gnu-gcc /g' Makefile
make
make install
cd ..

#-----librtmp 2.4 static lib
wget http://download.serviio.org/opensource/rtmpdump-c58cfb3e9208c6e6bc1aa18f1b1d650d799084e5.tar.gz
tar xvfz rtmpdump-c58cfb3e9208c6e6bc1aa18f1b1d650d799084e5.tar.gz
cd rtmpdump
#-----move all static libs to a separate folder to force compiler to use them
#-----http://www.gossamer-threads.com/lists/apache/dev/265052
mkdir /tmp/lib
cp /usr/local/x86_64-linux-gnu/lib/*.a /tmp/lib
#-----fix Makefile (won't compile without libdl linked)
#-----http://forum.luahub.com/index.php?topic=2390.0
sed -i.bak -e '/^LIB_OPENSSL\=/s/lcrypto/lcrypto \-ldl/' Makefile
make CROSS_COMPILE=x86_64-linux-gnu- SYS=posix prefix=/usr/local/x86_64-linux-gnu INC=-I/usr/local/x86_64-linux-gnu XLDFLAGS=-L/tmp/lib SHARED=
make install prefix=/usr/local/x86_64-linux-gnu SHARED=
cd ..
#-----gather all static libs again ready for FFmpeg compile
cp /usr/local/x86_64-linux-gnu/lib/*.a /tmp/lib
#-----remove unsupported URL line from pkgconfig/librtmp.pc
#-----pkg-config --exists --print-errors librtmp
sed -i -e '/^URL/d' /usr/local/x86_64-linux-gnu/lib/pkgconfig/librtmp.pc
export PKG_CONFIG_PATH="${PKG_CONFIG_PATH}:/usr/local/x86_64-linux-gnu/lib/pkgconfig"

#-----FFmpeg 0.8.2 (slightly newer than the one at Serviio.org)
wget http://dl.dropbox.com/u/1188556/ffmpeg-HEAD-05a2673.tar.gz
tar xvfz ffmpeg-HEAD-05a2673.tar.gz
cd ffmpeg-HEAD-05a2673
#-----Intel CPU-specific options
#-----pkg-config needs to be defined because with cross-prefix it assumes x86_64-linux-gnu-pkg-config which doesn't exist, and then librtmp won't be detected
sudo apt-get install yasm
./configure --arch=x86_64 --enable-ssse3 --cross-prefix=x86_64-linux-gnu- --target-os=linux --prefix=/usr/local/x86_64-linux-gnu
--extra-cflags='-I/usr/local/x86_64-linux-gnu/include' --extra-ldflags='-L/tmp/lib' --enable-static --disable-shared --disable-ffplay
--disable-ffserver --enable-pthreads --enable-libmp3lame --enable-librtmp --pkg-config=pkg-config --extra-version=Serviio
make
make install
 

Serviio 0.6 package for Synology NAS with Web UI

Serviio-package-with-webUI

The Serviio Web UI, styled to look like the regular DSM control panel widgets

 

UPDATE – This is no longer the most recent package. I have released a new version and created a new blog post to keep the comments on topic.

My guide for installing Petr Nejedly’s excellent Serviio 0.5.2 on Synology has been a very popular post, and many people have got it running. However, the process is still complex enough to deter a lot of people and, as I learned more about my Synology NAS, I realised that I could automate the whole thing. The biggest obstacle was creating a package for Java, but adding locale support to the syno Linux also proved difficult.

The PHP Web UI is now included which you can use to manage Serviio once it’s installed. I have tried to fix most of the localization issues with it, and have edited the CSS to give it the look and feel of DSM as much as I could. You could also use Serviio Remote Console from another computer if you prefer. Any online content plugins (ending in .groovy) that are found in your NAS ‘public’ share will also be installed into Serviio’s plugins folder automatically during installation.

The package will allow future upgrades while preserving the media database and the online plugins, as long as Serviio itself is ok with that. You can see Serviio’s log by clicking More in the Package Centre. DSM Package Center installs the application to /volume1/@appstore/Serviio though from what I understand, on multi-volume systems the user is prompted for a destination volume. If you need to edit device profiles, add online content plugins, or look at the log in detail that’s where you’ll find the files. Bear in mind that a NAS, even an Intel Atom powered one, is unlikely to have sufficient CPU power to manage realtime video transcoding (e.g. from DivX to MPEG2), though it will manage remuxing MKV to M2TS and realtime audio transcoding.

 

Download

If you have DSM 3.2, you can install directly from Package Center. In Settings -> Package Sources add my package repository URL which is http://packages.pcloadletter.co.uk

If you’re still on DSM 3.1, download one of the following packages. Right-click, Save As:

Serviio uses some open source libraries without which it would not have been possible.
By downloading the software on this page you are agreeing to the terms of the original Serviio End User Licence Agreement.
I accept no liability for any damage that may occur to your data or your cat’s health as a result of using this software :) The scripts are shown in full below, so you are free to assess any risks yourself.

 

Instructions

This package can be installed on an unmodified NAS – no hacking is required. It has been tested on DSM 3.1 and DSM 3.2. Here’s what you need to do:

  • In the DSM User control panel enable the User Home service
  • In the DSM Web Services control panel enable Web Station
  • Using the DSM Package Center install my Java SE for Embedded Synology package
  • Download any online content plugins that you require and leave the .groovy file for each in your NAS’s public shared folder
  • Determine your NAS’s CPU type, and install the appropriate Serviio Synology package from the download link above
  • Give it some time to start (click More Info in Package Center to see the Log tab – check it has loaded all the renderer profiles), then you can use the Serviio icon in the DSM drop down menu in the top-right to launch the Web UI. This Serviio icon will only be visible to admin users in DSM
  • To use the Serviio Remote Console from a Windows PC edit ServiioConsole.exe.vmoptions in the Serviio folder and add the line: -Dserviio.remoteHost=192.168.x.x (replace with your NAS’s IP address). It can take a minute or so to start – this is normal. For details on how to configure the Remote Console on other operating systems refer to the Serviio FAQ. It seems that the first time Serviio starts with a blank database the Remote Console will get stuck showing the message “Serviio is updating its Media Library”. If that happens wait a couple of minutes, stop and restart Serviio via the Package Center. Give it another couple of minutes then the remote console should work fine.
  • Once the console is up, disable ‘Generate thumbnails for local videos’, and set the ‘Transcoded files location’ to /volume1/@tmp. Don’t forget to use the Users control panel in DSM to grant permission for the Serviio user to read your files (if they’re not in the public share)
  • When you add folders to your library you need to use the ‘Add path…’ button (not ‘Add local…’) and give the full Linux path (e.g. /volume1/share-name/Movies). There are issues with the Add Local ajax file browser which mean that it doesn’t work properly even if you give PHP access to the filesystem root
 

The DSM main drop-down menu, showing the Serviio Web UI icon

 

Upgrading from 0.5.2

If you want to upgrade from a manually installed 0.5.2, you will need to follow the steps below. I was considering automating it, but it gets complicated trying to remove people’s edits from /etc/profile which may not be entirely consistent.

  • Stop the Serviio 0.5.2 daemon by running: /opt/etc/init.d/S99serviio.sh stop
  • Copy your media database somewhere safe: cp -R /opt/serviio/library/db /volume1/public
  • Follow the uninstallation steps at the bottom of my original guide
  • Install my Java Synology package
  • Install the Serviio Synology package from the download link above but don’t start it yet
  • Copy your saved database into position: cp -R /volume1/public/db /volume1/@appstore/Serviio/library (warning – for people with multiple volumes, the @appstore folder is not always on /volume1 apparently)
  • Force ownership on database files: chown -R serviio /volume1/@appstore/Serviio/library
  • Force ownership on transcode cache: chown -R serviio /volume1/@tmp/Serviio
  • Start Serviio in the DSM Package Center
  • Follow the rest of the settings guidelines from above
 

Package scripts

For info, here are the scripts inside the package. The security-conscious among you can untar the package and take a look at the contents for yourself.

preinst

#!/bin/sh

#--------SERVIIO preinstall script
#--------package maintained at pcloadletter.co.uk

source /etc/profile

if [ -z ${JAVA_HOME} ]; then
        echo "Java is not installed or not properly configured. JAVA_HOME is not defined. " > $SYNOPKG_TEMP_LOGFILE
        echo "Download and install the Java Synology package from http://wp.me/pVshC-z5" >> $SYNOPKG_TEMP_LOGFILE
        exit 1
fi

if [ ! -f ${JAVA_HOME}/bin/java ]; then
        echo "Java is not installed or not properly configured. The Java binary could not be located. " > $SYNOPKG_TEMP_LOGFILE
        echo "Download and install the Java Synology package from http://wp.me/pVshC-z5" >> $SYNOPKG_TEMP_LOGFILE
        exit 1
fi

exit 0
 

postinst

#!/bin/sh

#--------SERVIIO postinstall script
#--------package maintained at pcloadletter.co.uk

#create serviio daemon user
synouser --add serviio `${SYNOPKG_PKGDEST}/bin/passgen 1 20` "Serviio daemon user" 0 "" ""

#determine the serviio user homedir and save that variable in the user's profile
#this is needed because librtmp needs to write a file called ~/.swfinfo
#and new users seem to inherit a HOME value of /root which they have no permissions for
SERVIIO_HOMEDIR=`cat /etc/passwd | sed -r '/Serviio daemon user/!d;s/^.*:Serviio daemon user:(.*):.*$/\1/'`
su - serviio -s /bin/sh -c "echo export HOME=${SERVIIO_HOMEDIR} >> .profile"

#change owner of Serviio folder tree
chown -R serviio ${SYNOPKG_PKGDEST}

#reset ownership on any pre-existing transcoding temp folder (same volume as appstore)
if [ -d ${SYNOPKG_PKGDEST}/../../@tmp/Serviio ]; then
        chown -R serviio ${SYNOPKG_PKGDEST}/../../@tmp/Serviio
fi
#reset ownership on any pre-existing transcoding temp folder (on volume1 in case app was installed on a different volume)
if [ -d /volume1/@tmp/Serviio ]; then
        chown -R serviio /volume1/@tmp/Serviio
fi

#create symbolic link to Serviio's version of FFmpeg
if [ -f /opt/bin/ffmpeg ]; then
        mv /opt/bin/ffmpeg /opt/bin/ffmpeg.bak.serviio
        ln -s ${SYNOPKG_PKGDEST}/bin/ffmpeg /opt/bin/ffmpeg
else
        ln -s ${SYNOPKG_PKGDEST}/bin/ffmpeg /bin/ffmpeg
fi

#set up symlink for the DSM GUI
if [ -d /usr/syno/synoman/webman/3rdparty ]; then
        ln -s ${SYNOPKG_PKGDEST}/DSM /usr/syno/synoman/webman/3rdparty/Serviio
fi

#move web UI files to Webstation root folder
mv ${SYNOPKG_PKGDEST}/web /volume1/web/serviio

#find path of public share (it might not be on /volume1)
PLUGIN_PICKUP=`cat /usr/syno/etc/smb.conf | sed -r '/\/public$/!d;s/^.*path=(\/volume[0-9]{1,3}\/public).*$/\1/'`

#pick up any plugins from the public share
mv ${PLUGIN_PICKUP}/*.groovy ${SYNOPKG_PKGDEST}/plugins

exit 0
 

start-stop-status

#!/bin/sh

#--------SERVIIO start-stop-status script
#--------package maintained at pcloadletter.co.uk

daemon_status ()
{
        ps | grep '^ *[0-9]* serviio' > /dev/null
}

case $1 in
        start)
                #start Serviio in background mode
                su - serviio -s /bin/sh -c "${SYNOPKG_PKGDEST}/bin/serviio.sh &"
                exit 0

        ;;
        stop)
                su - serviio -s /bin/sh -c "${SYNOPKG_PKGDEST}/bin/serviio.sh -stop"
                exit 0
        ;;
        status)
                if daemon_status ; then
                        exit 0
                else
                        exit 1
                fi
        ;;
        log)
        echo "${SYNOPKG_PKGDEST}/log/serviio.log"
                exit 0
        ;;
esac
 

preuninst

#!/bin/sh

#--------SERVIIO preuninstall script
#--------package maintained at pcloadletter.co.uk

#make sure that Serviio is stopped
${SYNOPKG_PKGDEST}/bin/serviio.sh -stop
sleep 6

exit 0
 

postuninst

#!/bin/sh

#--------SERVIIO postuninstall script
#--------package maintained at pcloadletter.co.uk

#remove serviio daemon user
synouser --del serviio

#remove symlink and reinstate any backed up version of FFmpeg
if [ -L /opt/bin/ffmpeg ] && [ -f /opt/bin/ffmpeg.bak.serviio ]; then
        rm /opt/bin/ffmpeg
        mv /opt/bin/ffmpeg.bak.serviio /opt/bin/ffmpeg
fi
if [ -L /bin/ffmpeg ]; then
        rm /bin/ffmpeg
fi

#remove DSM icon symlink
rm /usr/syno/synoman/webman/3rdparty/Serviio

#remove web UI
if [ -d /volume1/web/serviio ]; then
        rm -r /volume1/web/serviio
fi

exit 0
 

preupgrade

#!/bin/sh

#--------SERVIIO preupgrade script
#--------package maintained at pcloadletter.co.uk

#make sure that Serviio is stopped
${SYNOPKG_PKGDEST}/bin/serviio.sh -stop
sleep 6

#if a media database exists we need to preserve it
if [ -d ${SYNOPKG_PKGDEST}/library/db ]; then
        mkdir ${SYNOPKG_PKGDEST}/../serviio_db_migration
        mv ${SYNOPKG_PKGDEST}/library/db ${SYNOPKG_PKGDEST}/../serviio_db_migration
fi

#if online plugins exist we need to migrate them
if ls ${SYNOPKG_PKGDEST}/plugins/*.groovy; then
        mkdir ${SYNOPKG_PKGDEST}/../serviio_plugin_migration
        mv ${SYNOPKG_PKGDEST}/plugins ${SYNOPKG_PKGDEST}/../serviio_plugin_migration
fi

exit 0
 

postupgrade

#!/bin/sh

#--------SERVIIO postupgrade script
#--------package maintained at pcloadletter.co.uk

#use the backed up media database from the previous version
if [ -d ${SYNOPKG_PKGDEST}/../serviio_db_migration/db ]; then
        mv ${SYNOPKG_PKGDEST}/../serviio_db_migration/db ${SYNOPKG_PKGDEST}/library
        rmdir ${SYNOPKG_PKGDEST}/../serviio_db_migration

        #serviio daemon user has been deleted and recreated so we need to reset ownership (new UID)
        chown -R serviio ${SYNOPKG_PKGDEST}/library/db
fi

#use the backed up plugins from the previous version
if [ -d ${SYNOPKG_PKGDEST}/../serviio_plugin_migration/plugins ]; then
        cp ${SYNOPKG_PKGDEST}/../serviio_plugin_migration/plugins/*.groovy ${SYNOPKG_PKGDEST}/plugins
        if [ ! -z ${SYNOPKG_PKGDEST} ]; then
                if [ -d ${SYNOPKG_PKGDEST}/../serviio_plugin_migration ]; then
                        rm -r ${SYNOPKG_PKGDEST}/../serviio_plugin_migration
                fi
        fi
        #serviio daemon user has been deleted and recreated so we need to reset ownership (new UID)
        chown -R serviio ${SYNOPKG_PKGDEST}/plugins
fi

#reset ownership on transcoding temp folder (same volume as appstore)
if [ -d ${SYNOPKG_PKGDEST}/../../@tmp/Serviio ]; then
        chown -R serviio ${SYNOPKG_PKGDEST}/../../@tmp/Serviio
fi
#reset ownership on transcoding temp folder (on volume1 in case app was installed on a different volume)
if [ -d /volume1/@tmp/Serviio ]; then
        chown -R serviio /volume1/@tmp/Serviio
fi

#restart Web Station to clear webserver caches
if [ -f /usr/syno/etc/rc.d/S97apache-user.sh ]; then
        /usr/syno/etc/rc.d/S97apache-user.sh restart
fi

exit 0
 

Changelog:

  • 0.6.0.1-005 added Web Station dependency, EULA dialog, and links for Web UI and user forum in More Info
  • 0.6.0.1-004 test for package repo to allow update notification
  • 0.6.0.1 v3 hopefully fixed an issue with the Serviio DSM icon in DSM 3.2 on NAS units without Optware installed
  • 0.6.0.1 v2 inclusion of a modified version of the PHP Web UI
  • 0.6.0.1 v1 new minor Serviio release with some updated international translations – will upgrade 0.6 preserving the media library
  • 0.6 v1 first public release
  • 0.6b4 v2 fixed permissions on transcode temporary folder following upgrade, avoiding the need for a reboot
  • 0.6b4 v1 added upgrade scripts to allow media database migration (install future packages over the top of this one – database is preserved)
  • 0.6b3c v2 fixed HOME env var for serviio user (to fix librtmp issues with BBC iPlayer) and removed the need to edit the passwd file to change the shell (safer)
  • 0.6b3c v1 initial spk test release

 
 

Java SE Embedded package for Synology NAS

There is a lot of good Java software, but it’s difficult to get Java running on a NAS. I decided to do something about that, mainly to pave the way for a Serviio package that didn’t require a huge guide to install. In compliance with the Oracle EULA the Java binaries cannot be redistributed, so this package looks for the downloaded .tar.gz archive in the public shared folder on the NAS. The end user is required to register with Oracle to get the file, as you can see from the description below. I also added locale support to the underlying Linux since Java requires this for UTF-8 support when reading from the filesystem (DSM 4.3 later included this). Timezone support is also configured.

This package installs Java SE Embedded which is purposefully designed to run on low specification headless hardware. It should be noted that Synology Java Manager (only available on Intel products) is the full Java Runtime Environment, and consequently uses more RAM. By default you should use Java 8, unless the application you want to run specifically requires Java 7.

Please note that Oracle ceased maintaining Java 6 in 2012, so it should be considered a serious security vulnerability. The Java 6 package has been withdrawn from the package repository for this reason. If you need it to run a particular piece of software you may download it here for manual installation. Please also note that Oracle retired support for PowerPC CPUs following Java 8 Update 6, so the PowerPC package remains at that version.

Java Package

 

Instructions

  • In Synology DSM’s Package Center, click Settings and add my package repository:
    Add Package Repository
  • The repository will push its certificate automatically to the NAS, which is used to validate package integrity. Set the Trust Level to Synology Inc. and trusted publishers:
    Trust Level
  • Now browse the Community section in Package Center to install a Java SE Embedded package:
    Community-packages
    The repository only displays packages which are compatible with your specific model of NAS. If you don’t see Java Embedded in the list, then either your NAS model or your DSM version are not supported at this time.
  • If you have a multi-bay NAS, use the Shared Folder control panel to create a shared folder called public (it must be all lower case). On single bay models this is created by default. Assign it with Read/Write privileges for everyone.
  • Register with Oracle and download the appropriate version of Java SE Embedded for your CPU architecture and save in the public shared folder on your NAS. Note that to get any version of Java older than the current one you will need to follow the Oracle Java SE Embedded Downloads Archive link on that page. If in doubt, attempt to install the package first, and the error message will tell you which specific download version is required.
  • If you have trouble getting the Java archive recognised, try downloading it with a different web browser. Some browsers try to help by uncompressing the file, or renaming it without warning. I have tried to code around most of these behaviours. Use Firefox if all else fails.
 

Package scripts

For information, here are the package scripts so you can see what it’s going to do. You can get more information about how packages work by reading the Synology 3rd Party Developer Guide.

installer.sh

#!/bin/sh

#--------JAVA installer script
#--------package maintained at pcloadletter.co.uk

COMMENT="# Synology Java Package"
SYNO_CPU_ARCH="`uname -m`"
[ "`echo ${SYNO_CPU_ARCH} | cut -c1-7`" == "armv5te" ] && SYNO_CPU_ARCH="armv5tel"
#--------Synology switched Armada 370 systems from SoftFP to HardFP EABI for DSM 6.0
[ "${SYNOPKG_DSM_ARCH}" == "armada370" ] && [ ${SYNOPKG_DSM_VERSION_MAJOR} -gt 5 ] && SYNO_CPU_ARCH="armv7l-hflt"
[ "${SYNOPKG_DSM_ARCH}" == "armada375" ] && SYNO_CPU_ARCH="armv7l-hflt"
[ "${SYNOPKG_DSM_ARCH}" == "armada38x" ] && SYNO_CPU_ARCH="armv7l-hflt"
[ "${SYNOPKG_DSM_ARCH}" == "comcerto2k" ] && SYNO_CPU_ARCH="armv7l-hflt"
[ "${SYNOPKG_DSM_ARCH}" == "alpine" ] && SYNO_CPU_ARCH="armv7l-hflt"
[ "${SYNOPKG_DSM_ARCH}" == "alpine4k" ] && SYNO_CPU_ARCH="armv7l-hflt"
[ "${SYNOPKG_DSM_ARCH}" == "monaco" ] && SYNO_CPU_ARCH="armv7l-hflt"
[ "${SYNOPKG_DSM_ARCH}" == "x86_64" ] && [ "${WIZARD_JRE_EE}" == "true" ] && SYNO_CPU_ARCH="i686"
[ "${SYNOPKG_DSM_ARCH}" == "x86_64" ] && [ ${SYNOPKG_DSM_VERSION_MAJOR} -lt 6 ] && SYNO_CPU_ARCH="i686"

if [ "${SYNOPKG_PKGNAME}" == "java6" ]; then
  DOWNLOAD_URL="http://tinyurl.com/javaembedarchive"
  EXTRACTED_FOLDER="ejre1.6.0_38"
  JCE_BINARY="jce_policy-6.zip"
  JCE_URL="http://tinyurl.com/java-jce6"
  JCE_FOLDER="jce"
  if [ "`echo ${SYNO_CPU_ARCH} | cut -c1-7`" == "armv5te" ]; then
    JAVA_BINARY="ejre-1_6_0_38-fcs-b05-linux-arm-sflt-eabi-headless-13_nov_2012.tar.gz"
    JAVA_BUILD="ARMv5 Linux - Headless EABI"
  elif [ "${SYNO_CPU_ARCH}" == "armv7l" ]; then
    JAVA_BINARY="ejre-1_6_0_38-fcs-b05-linux-arm-vfp-eabi-headless-13_nov_2012.tar.gz"
    JAVA_BUILD="ARMv6/7 Linux - Headless EABI, VFP, SoftFP ABI, Little Endian"
  elif [ "${SYNO_CPU_ARCH}" == "i686" ]; then
    JAVA_BINARY="ejre-1_6_0_38-fcs-b05-linux-i586-headless-13_nov_2012.tar.gz"
    JAVA_BUILD="x86 Linux Small Footprint - Headless"
  elif [ "${SYNO_CPU_ARCH}" == "x86_64" ]; then
    JAVA_BINARY="ejre-1_6_0_38-fcs-b05-linux-i586-headless-13_nov_2012.tar.gz"
    JAVA_BUILD="x86 Linux Small Footprint - Headless"
  elif [ "${SYNO_CPU_ARCH}" == "ppc" ]; then
    JAVA_BINARY="ejre-1_6_0_38-fcs-b05-linux-ppc-e500v2-headless-13_nov_2012.tar.gz"
    JAVA_BUILD="Power Architecture Linux - Headless - e500v2 core"
  fi    

elif [ "${SYNOPKG_PKGNAME}" == "java7" ]; then
  DOWNLOAD_URL="http://tinyurl.com/javaembed"
  EXTRACTED_FOLDER="ejre1.7.0_75"
  JCE_BINARY="UnlimitedJCEPolicyJDK7.zip"
  JCE_URL="http://tinyurl.com/java-jce7"
  JCE_FOLDER="UnlimitedJCEPolicy"
  if [ "${SYNO_CPU_ARCH}" == "armv5tel" ]; then
    JAVA_BINARY="ejre-7u75-fcs-b13-linux-arm-sflt-headless-18_dec_2014.tar.gz"
    JAVA_BUILD="ARMv5 Linux - Headless EABI, SoftFP ABI, Little Endian"
  elif [ "${SYNO_CPU_ARCH}" == "armv7l" ]; then
    JAVA_BINARY="ejre-7u75-fcs-b13-linux-arm-vfp-sflt-client_headless-18_dec_2014.tar.gz"
    JAVA_BUILD="ARMv6/7 Linux - Headless - Client Compiler EABI, VFP, SoftFP ABI, Little Endian"
  elif [ "${SYNO_CPU_ARCH}" == "armv7l-hflt" ]; then
    JAVA_BINARY="ejre-7u75-fcs-b13-linux-arm-vfp-hflt-client_headless-18_dec_2014.tar.gz"
    JAVA_BUILD="ARMv6/7 Linux - Headless - Client Compiler EABI, VFP, HardFP ABI, Little Endian"
  elif [ "${SYNO_CPU_ARCH}" == "i686" ]; then
    JAVA_BINARY="ejre-7u75-fcs-b13-linux-i586-headless-18_dec_2014.tar.gz"
    JAVA_BUILD="x86 Linux Small Footprint - Headless"  
  elif [ "${SYNO_CPU_ARCH}" == "x86_64" ]; then
    JAVA_BINARY="ejre-7u75-fcs-b13-linux-i586-headless-18_dec_2014.tar.gz"
    JAVA_BUILD="x86 Linux Small Footprint - Headless"  
  elif [ "${SYNO_CPU_ARCH}" == "ppc" ]; then
    JAVA_BINARY="ejre-7u75-fcs-b13-linux-ppc-e500v2-headless-18_dec_2014.tar.gz"
    JAVA_BUILD="Power Architecture Linux - Headless - e500v2 with double-precision SPE Floating Point Unit"
  fi

elif [ "${SYNOPKG_PKGNAME}" == "java8" ]; then
  DOWNLOAD_URL="http://tinyurl.com/javaembed"
  EXTRACTED_FOLDER="ejdk1.8.0_211"
  JCE_BINARY="jce_policy-8.zip"
  JCE_URL="http://tinyurl.com/java-jce8"
  JCE_FOLDER="UnlimitedJCEPolicyJDK8"
  if [ "${SYNO_CPU_ARCH}" == "armv5tel" ]; then
    JAVA_BINARY="ejdk-8u211-linux-arm-sflt.tar.gz"
    JAVA_BUILD="ARMv5/ARMv6/ARMv7 Linux - SoftFP ABI, Little Endian 2"
  elif [ "${SYNO_CPU_ARCH}" == "armv7l" ]; then
    JAVA_BINARY="ejdk-8u211-linux-arm-sflt.tar.gz"
    JAVA_BUILD="ARMv5/ARMv6/ARMv7 Linux - SoftFP ABI, Little Endian 2"
  elif [ "${SYNO_CPU_ARCH}" == "armv7l-hflt" ]; then
    JAVA_BINARY="ejdk-8u211-linux-armv6-vfp-hflt.tar.gz"
    JAVA_BUILD="ARMv6/ARMv7 Linux - VFP, HardFP ABI, Little Endian 1"
  elif [ "${SYNO_CPU_ARCH}" == "aarch64" ]; then
    JAVA_BINARY="jdk-8u211-linux-arm64-vfp-hflt.tar.gz"
    JAVA_BUILD="Linux ARM 64 Hard Float ABI"
    DOWNLOAD_URL="http://tinyurl.com/java8arm64"
    EXTRACTED_FOLDER="jdk1.8.0_211"
  elif [ "${SYNO_CPU_ARCH}" == "i686" ]; then
    JAVA_BINARY="ejdk-8u211-linux-i586.tar.gz"
    JAVA_BUILD="x86 Linux Small Footprint - Headless"
  elif [ "${SYNO_CPU_ARCH}" == "x86_64" ]; then
    JAVA_BINARY="jre-8u211-linux-x64.tar.gz"
    JAVA_BUILD="Linux x64"
    EXTRACTED_FOLDER="jre1.8.0_211"
    DOWNLOAD_URL="http://tinyurl.com/java8x64"
  elif [ "${SYNO_CPU_ARCH}" == "ppc" ]; then
    #Oracle have discontinued Java 8 for PowerPC after update 6
    JAVA_BINARY="ejdk-8u6-fcs-b23-linux-ppc-e500v2-12_jun_2014.tar.gz"
    JAVA_BUILD="Power Architecture Linux - Headless - e500v2 with double-precision SPE Floating Point Unit"
    EXTRACTED_FOLDER="ejdk1.8.0_06"
  fi
fi

JAVA_BINARY=`echo ${JAVA_BINARY} | cut -f1 -d'.'`
PUBLIC_FOLDER="`synoshare --get public | sed -r "/Path/!d;s/^.*\[(.*)\].*$/\1/"`"
TEMP_FOLDER="`find / -maxdepth 2 -path '/volume?/@tmp' | head -n 1`"
NATIVE_BINS_URL="http://packages.pcloadletter.co.uk/downloads/java-native-${SYNO_CPU_ARCH}.tar.xz"
NATIVE_BINS_FILE="`echo ${NATIVE_BINS_URL} | sed -r "s%^.*/(.*)%\1%"`"
#DSM versions older than 4.3 need locale support adding, don't download unless needed
[ ! -e /usr/bin/locale ] && INSTALL_FILES="${NATIVE_BINS_URL}"
source /etc/profile


preinst ()
{
  synoshare -get public > /dev/null || (
    echo "A shared folder called 'public' could not be found - note this name is case-sensitive. " >> $SYNOPKG_TEMP_LOGFILE
    echo "Please create this using the Shared Folder DSM Control Panel and try again." >> $SYNOPKG_TEMP_LOGFILE
    exit 1
  )

  JAVA_BINARY_FOUND=

  if [ -n "${JAVA_HOME}" ]; then
    echo "It seems from /etc/profile that a Java Runtime is already installed at ${JAVA_HOME}. Uninstall it and try again." >> $SYNOPKG_TEMP_LOGFILE
    exit 1
  fi
  
  [ -f ${PUBLIC_FOLDER}/${JAVA_BINARY}.tar.gz ] && JAVA_BINARY_FOUND=true
  [ -f ${PUBLIC_FOLDER}/${JAVA_BINARY}.tar ] && JAVA_BINARY_FOUND=true
  [ -f ${PUBLIC_FOLDER}/${JAVA_BINARY}.tar.tar ] && JAVA_BINARY_FOUND=true
  [ -f ${PUBLIC_FOLDER}/${JAVA_BINARY}.gz ] && JAVA_BINARY_FOUND=true
  
  if [ -z ${JAVA_BINARY_FOUND} ]; then
    echo "Java binary bundle not found. " >> $SYNOPKG_TEMP_LOGFILE
    echo "I was expecting the file ${PUBLIC_FOLDER}/${JAVA_BINARY}.tar.gz. " >> $SYNOPKG_TEMP_LOGFILE
    echo "Please agree to the Oracle licence at ${DOWNLOAD_URL}, then download the '${JAVA_BUILD}' package" >> $SYNOPKG_TEMP_LOGFILE
    echo "and place it in the 'public' shared folder on your NAS. This download cannot be automated even if " >> $SYNOPKG_TEMP_LOGFILE
    echo "displaying a package EULA could potentially cover the legal aspect, because files hosted on Oracle's " >> $SYNOPKG_TEMP_LOGFILE
    echo "server are protected by a session cookie requiring a JavaScript enabled browser." >> $SYNOPKG_TEMP_LOGFILE
    exit 1
  fi

  if [ "${WIZARD_JCE_UL}" == "true" ]; then
    if [ ! -f ${PUBLIC_FOLDER}/${JCE_BINARY} ]; then
      echo "JCE Unlimited Strength binary bundle not found. " >> $SYNOPKG_TEMP_LOGFILE
      echo "I was expecting the file ${PUBLIC_FOLDER}/${JCE_BINARY} " >> $SYNOPKG_TEMP_LOGFILE
      echo "Please agree to the Oracle licence at ${JCE_URL}, then download the Unlimited Strength Jurisdiction Policy Files zip file " >> $SYNOPKG_TEMP_LOGFILE
      echo "and place it in the 'public' shared folder on your NAS. " >> $SYNOPKG_TEMP_LOGFILE
      exit 1
    fi    
  fi    

  cd ${TEMP_FOLDER}
  for WGET_URL in ${INSTALL_FILES}
  do
    WGET_FILENAME="`echo ${WGET_URL} | sed -r "s%^.*/(.*)%\1%"`"
    [ -f ${TEMP_FOLDER}/${WGET_FILENAME} ] && rm ${TEMP_FOLDER}/${WGET_FILENAME}
    wget ${WGET_URL}
    if [[ $? != 0 ]]; then
      if [ -d ${PUBLIC_FOLDER} ] && [ -f ${PUBLIC_FOLDER}/${WGET_FILENAME} ]; then
        cp ${PUBLIC_FOLDER}/${WGET_FILENAME} ${TEMP_FOLDER}
      else     
        echo "There was a problem downloading ${WGET_FILENAME} from the official download link, " >> $SYNOPKG_TEMP_LOGFILE
        echo "which was \"${WGET_URL}\" " >> $SYNOPKG_TEMP_LOGFILE
        echo "Alternatively, you may download this file manually and place it in the 'public' shared folder. " >> $SYNOPKG_TEMP_LOGFILE
        exit 1
      fi
    fi
  done
  
  exit 0
}


postinst ()
{
  WORKING_DIR=`pwd`
  
  #ldd tool is very useful but not included in DSM
  if [ ! -e /usr/local/bin/ldd ]; then
    mkdir -p /usr/local/bin
    cp ${SYNOPKG_PKGDEST}/tools/ldd-${SYNO_CPU_ARCH} /usr/local/bin/ldd
  fi

  #extract native binaries if needed
  if [ -e ${TEMP_FOLDER}/${NATIVE_BINS_FILE} ]; then
    cd ${SYNOPKG_PKGDEST}
    #older DSM tar uses different command line switch for xz archives
    tar xJf ${TEMP_FOLDER}/${NATIVE_BINS_FILE} || tar xaf ${TEMP_FOLDER}/${NATIVE_BINS_FILE}
    rm ${TEMP_FOLDER}/${NATIVE_BINS_FILE}
  fi

  #extract Java (Web browsers love to interfere with .tar.gz files)
  cd ${PUBLIC_FOLDER}
  if [ -f ${JAVA_BINARY}.tar.gz ]; then
    #Firefox seems to be the only browser that leaves it alone
    tar xzf ${JAVA_BINARY}.tar.gz
  elif [ -f ${JAVA_BINARY}.gz ]; then
    #Chrome
    tar xzf ${JAVA_BINARY}.gz
  elif [ -f ${JAVA_BINARY}.tar ]; then
    #Safari
    tar xf ${JAVA_BINARY}.tar
  elif [ -f ${JAVA_BINARY}.tar.tar ]; then
    #Internet Explorer
    tar xzf ${JAVA_BINARY}.tar.tar
  fi

  #install Java
  if [ "${EXTRACTED_FOLDER}" != "${EXTRACTED_FOLDER/jdk/}" ]; then
    mv ${EXTRACTED_FOLDER} ${SYNOPKG_PKGDEST}
    JRE_PATH="`find ${SYNOPKG_PKGDEST}/${EXTRACTED_FOLDER} -name jre`"
  else
    mv ${EXTRACTED_FOLDER} ${SYNOPKG_PKGDEST}/jre
    JRE_PATH="${SYNOPKG_PKGDEST}/jre"
  fi

  #install unlimited crypto policy files if requested
  if [ "${WIZARD_JCE_UL}" == "true" ]; then
    if [ -e /usr/bin/7z ]; then
      7z x ${JCE_BINARY} > /dev/null 2>&1
    else
      unzip ${JCE_BINARY} 
    fi
    mv ${JCE_FOLDER}/* ${JRE_PATH}/lib/security
    rmdir ${JCE_FOLDER}
  fi  

  #change owner of folder tree
  chown -R root:root ${SYNOPKG_PKGDEST}

  echo "NOTE - This package does *not* start and stop like other packages. " >> $SYNOPKG_TEMP_LOGFILE
  echo "Java is correctly installed if you can see the runtime and HotSpot version numbers, " >> $SYNOPKG_TEMP_LOGFILE
  echo "and locale information in the package Log tab." >> $SYNOPKG_TEMP_LOGFILE
  
  exit 0
}


preuninst ()
{
  exit 0
}


postuninst ()
{
  #clean up profile mods
  sed -i "/${COMMENT}/d" /etc/profile
  sed -i "/${COMMENT}/d" /root/.profile

  #leave locale support in place on older DSM versions - too risky to delete system binaries
  
  exit 0
}
 

start-stop-status.sh

#!/bin/sh

JRE_PATH="`find /var/packages/${SYNOPKG_PKGNAME}/target/ -name jre`"
COMMENT="# Synology Java Package"
#set the current timezone for Java so that log timestamps are accurate, modern timezone names so DST works
SYNO_TZ=`cat /etc/synoinfo.conf | grep timezone | cut -f2 -d'"'`
#fix for DST time in DSM 5.2 thanks to MinimServer Syno package author
[ -e /usr/share/zoneinfo/Timezone/synotztable.json ] \
 && SYNO_TZ=`jq ".${SYNO_TZ} | .nameInTZDB" /usr/share/zoneinfo/Timezone/synotztable.json | sed -e "s/\"//g"` \
 || SYNO_TZ=`grep "^${SYNO_TZ}" /usr/share/zoneinfo/Timezone/tzname | sed -e "s/^.*= //"`

EnvCheck ()
#updates to DSM will reset these changes so check them each startup 
{
  #/etc/profile should contain 5 lines added by this package tagged with trailing comments
  COUNT=`grep -c "$COMMENT$" /etc/profile`
  if [ $COUNT != 5 ]; then

    #remove any existing mods
    sed -i "/${COMMENT}/d" /etc/profile

    #add required environment variables
    echo "PATH=\$PATH:${JRE_PATH}/bin ${COMMENT}" >> /etc/profile
    echo "JAVA_HOME=${JRE_PATH} ${COMMENT}" >> /etc/profile
    echo "CLASSPATH=.:${JRE_PATH}/lib ${COMMENT}" >> /etc/profile
    echo "LANG=en_US.utf8 ${COMMENT}" >> /etc/profile
    echo "export CLASSPATH JAVA_HOME LANG PATH ${COMMENT}" >> /etc/profile
  fi

  #/root/.profile should contain 3 lines added by this package tagged with trailing comments
  COUNT=`grep -c "$COMMENT$" /root/.profile`
  if [ $COUNT != 3 ]; then

    #remove any existing mods
    sed -i "/${COMMENT}/d" /root/.profile

    #add required environment variables
    echo "PATH=\$PATH:${JRE_PATH}/bin ${COMMENT}" >> /root/.profile
    echo "TZ='${SYNO_TZ}' ${COMMENT}" >> /root/.profile
    echo "export PATH TZ ${COMMENT}" >> /root/.profile
  fi

  #timezone needs setting each startup in case system value has changed
  sed -i "s%^TZ=.* ${COMMENT}%TZ='${SYNO_TZ}' ${COMMENT}%" /root/.profile

  #DSM versions older than 4.3 need locale support adding
  if [ ! -e /usr/bin/locale ]; then

    #build missing locale with UTF-8 support (don't think it matters which language)
    cp ${SYNOPKG_PKGDEST}/bin/* /usr/bin
    cp -R ${SYNOPKG_PKGDEST}/share/i18n /usr/share
    if [ ! -d /usr/lib/locale ]; then
      mkdir /usr/lib/locale
    fi
    localedef -c -f UTF-8 -i en_US en_US.utf8
  fi

  #DSM versions up to 5.1 are missing the zoneinfo timezone definitions which causes inaccurate log timestamps
  #thanks to CoolRaoul here: http://forum.synology.com/enu/viewtopic.php?f=19&t=69232#p310553
  #to revert, just remove everything apart from 'Timezone' 'zone.tab' from /usr/share/zoneinfo
  if [ ! -e /usr/share/zoneinfo/Europe/London ]; then
    cp -R ${SYNOPKG_PKGDEST}/zoneinfo/* /usr/share/zoneinfo/
  fi

}

case $1 in
  start)
    EnvCheck
    source /etc/profile
    source /root/.profile

    #evidence of whether Java can start successfully is written to the package log
    java -version > ${SYNOPKG_PKGDEST}/output.log 2>&1
    echo >> ${SYNOPKG_PKGDEST}/output.log
    echo System installed locales: >> ${SYNOPKG_PKGDEST}/output.log
    locale -a >> ${SYNOPKG_PKGDEST}/output.log
    echo >> ${SYNOPKG_PKGDEST}/output.log
    echo JAVA_HOME=$JAVA_HOME >> ${SYNOPKG_PKGDEST}/output.log
    echo TZ=$TZ >> ${SYNOPKG_PKGDEST}/output.log

    exit 0
  ;;

  stop)
    exit 0
  ;;

  status)
    source /etc/profile
    source /root/.profile
    if [ -e "${JAVA_HOME}/bin/java" ]; then
      exit 0
    else
      exit 1
    fi
  ;;

  log)
    echo "${SYNOPKG_PKGDEST}/output.log"
    exit 0
  ;;

esac
 

install_uifile

[
  {
    "step_title": "Java Runtime Environment Selection",
    "items": [
      {
        "type": "singleselect",
        "desc": "Please select the Java version to install:",
        "subitems": [
          {
            "key": "WIZARD_JRE_EE",
            "desc": "Java SE Embedded 8 (smaller memory footprint, 32bit)",
            "defaultValue": true
          },
          {
            "key": "WIZARD_JRE_SE",
            "desc": "Java SE 8 (64bit)",
            "defaultValue": false
          }
        ]
      }
    ]
  },
  {
    "step_title": "Java Cryptography Strength Selection",
    "items": [
      {
        "type": "singleselect",
        "desc": "Please select Java Cryptographic Extension policy:",
        "subitems": [
          {
            "key": "WIZARD_JCE_STD",
            "desc": "Default Strong JCE policy - limited to AES-128 encryption",
            "defaultValue": true
          },
          {
            "key": "WIZARD_JCE_UL",
            "desc": "Unlimited Strength JCE policy - not permitted in some jurisdictions, see Oracle documentation",
            "defaultValue": false
          }
        ]
      }
    ]
  }
]
 

Changelog:

  • 0050 17/Apr/19 – Updated to Java 8u211, added support for DS119j
  • 0049 11/Apr/19 – Updated to Java 8u201, fixed issue where x86_64 CPUs would ignore user selection and always install 64bit Java SE
  • 0048 22/Aug/18 – Updated to Java 8u181, added option to select 64bit full Java SE for x86_64 CPUs
  • 0047 14/May/18 – Updated to Java 8u171
  • 0046 20/Feb/18 – Updated to Java 8u161, experimental aarch64 support
  • 0045 30/Oct/17 – Updated to Java 8u151, added support for new CPUs in x17 and x18 models
  • 0044 02/Aug/17 – Updated to Java 8u144
  • 0043 21/Jan/17 – Updated to Java 8u121
  • 0042 07/Jan/17 – Updated to Java 8u111, fixed Armada 370/XP systems running DSM 6.0 which had switched from soft to hard float ABI
  • 0041 25/Jul/16 – Added support for unlimited strength cryptography (default is strong)
  • 0040 20/Jul/16 – Updated to Java 8u101
  • 0039 26/Apr/16 – Updated to Java 8u91
  • 0038 24/Mar/16 – Updated to Java 8u77
  • 0037 07/Feb/16 – Updated to Java 8u73
  • 0036 26/Jan/16 – Updated to Java 8u71
  • 0035 14/Dec/15 – Updated to Java 8u65, added support for models DS716+ and DS216play
  • 0034 19/Aug/15 – Fixed timezone DST issue with DSM 5.2, added support for DS215+
  • 0033 07/Aug/15 – Updated to Java 8u51, PowerPC build remains at 8u6
  • 0032 13/Jul/15 – Fixed timezone issue with DSM 5.2, updated main blog post documentation
  • 0031 25/Jan/15 – Fixed issue with version 0030 for systems with more than one volume
  • 0030 21/Jan/15 – Improved temp folder detection, updated to Java 7u75 and Java 8u33 (Oracle did not publish a version for PowerPC)
  • 0029 25/Dec/14 – DSM timezone zone.tab file is no longer overwritten, which had been prompting a malware alert in DSM 5.1 Security Advisor
  • 0028 20/Oct/14 – Updated to Java 7u71, added missing zoneinfo timezone definitions to fix inaccurate timestamps for non-GMT timezones (checked every startup), switched native binary archives from .tgz to .tar.xz format to reduce size
  • 0027 18/Oct/14 – Updated Java 8 to 8u6, added support for Mindspeed Comcerto 2000 CPU in DS414j for Java 7 and Java 8
  • 0026 24/Jun/14 – DSM updates will no longer break Java
  • 0025 31/May/14 – Updated to Java 7u60
  • 0024 31/May/14 – Updated to Java 7u55
  • 0023 27/May/14 – Added Java 8 support
  • 0022 30/Jan/14 – Updated to Java 7u51
  • 0021 15/Nov/13 – Locale support is no longer downloaded and added to DSM 4.3 or newer since it’s already present
  • 0020 30/Oct/13 – Added support for Intel Atom Evansport and Armada XP CPUs in new DSx14 products, removed Early Access JDK 8 support since it did not run on the Armada SoCs
  • 0019 17/Oct/13 – Updated to Java 7u45
  • 0018 11/Oct/13 – Updated to Java 7u40, and JDK8 Early Access b106
  • 0017 05/Jun/13 – For Armada370 CPU changed from ARMv7 Server JVM to Client JVM since the Server one turned out to be unstable. Added support for Early Access JDK 8 for Armada 370.
  • 0016 21/May/13 – Added Armada370 CPU support (DS213j)
  • 0015 18/Apr/13 – Updated to Java 7u21, Java 6 Embedded seems to be no longer maintained by Oracle
  • 0014 13/Feb/13 – Fixed metadata for DSM 4.2 Package Center
  • 013 13/Dec/12 – Updated to Java 6u38 and Java 7u10
  • 012 10/Dec/12 – Installer script fetches native binaries separately to reduce bandwidth on repo hosting
  • 011 Added support for Freescale QorIQ PowerPC CPUs used in some Synology x13 series products, PowerQUICC PowerPC processors in previous Synology generations with older glibc versions are not supported
  • 010 updated to Java 6u34 and Java 7u6
  • 009 corrected timezone bug for Atlantic and Pacific timezones which may have caused error messages when in fact the package had installed correctly
  • 008 unified the build scripts
  • 007 included the locale binaries to simplify installation, fixed environment variables for root user (inc. timezone with DST support), displayed Java version information and env vars in Log tab, incremented Java to latest versions 6u32 and 7u4, fixed detection of renamed .tar.gz archives downloaded by Google Chrome
  • 006 fixed path issue for root user on non-bootstrapped systems, updated info link to refer back to this page, and improved description text
  • 005 incremented JRE versions to match new releases from Oracle
  • 004 forced check for existence of ‘public’ shared folder
  • 003 fixed a stupid typo that stopped 002 from working :)
  • 002 rewrote scripts to be easier to maintain, and to allow for web browsers that untar or rename the Java and toolchain binary bundles as they’re saved
  • 001 23/Aug/11 – fixed package repo support, defined as a non-runnable service, prevented more than one JRE being installed, saved files into actual package folder rather than @appstore/java now that package names can be separate from Package Center display names
  • v3 initial spk release (I think!)
 
 

Serviio 0.6 DLNA server on Synology NAS

Serviio-Synology

Update – I have released a Synology package for Serviio 0.6

 

Serviio is an excellent free Java DLNA media server by Petr Nejedly which focuses on minimizing the amount of unnecessary media transcoding, and maximizing the use of renderer devices’ supported features. Version 0.6 adds support for streaming content from online RSS feeds. Since the installation process has changed a lot, I decided to create a new post rather than edit the old one. The guide is accurate for Synology DSM 3.2 beta, and the one tricky section with wget-ssl also applies for DSM 3.1. Older DSM versions may vary.

In the Synology DSM go to Control Panel > Terminal > enable SSH.

Read this Synology wiki document about modifying your NAS carefully and install the bootstrap for your model. If you have previously been messing around with your unit and want to set it back to defaults, you can run the bootstrap again. It will prompt you to delete a couple of folders and reboot, whereupon you can start afresh.

Download the PuTTY SSH client if you’re using Windows – other operating systems will already allow you to SSH from the terminal. Give the session a name, set the remote character set to UTF-8 as shown and save it before you connect (to save time in future). This will ensure that all characters display properly. The Mac OS SSH client defaults to UTF-8.
PuTTY remote character set configuration

Connect to your NAS’s IP address using SSH. Use the root account (same password as admin). For Mac OS for instance, use ssh root@x.x.x.x. I suggest enabling a more descriptive shell prompt which should reduce the chance of accidentally being in the wrong directory:

sed -i -r 's/(^PS1=.*$)/#\1\n\1/' ~/.profile
sed -i -e 's/^PS1=.*$/PS1=\"\\w\\\$ \"/g' ~/.profile
source ~/.profile
 

sed (stream editor) is a powerful tool which uses regular expressions. The first command opens the file ~./profile searches for the line beginning with PS1= and duplicates it, commenting out the first copy (so the change can be undone). Then second command sets the value PS1=”\w\$ “ which changes the prompt to be the current directory followed by dollar sign and space characters.

To install the development tools. Type:

ipkg update
ipkg install optware-devel
 

It will halt and complain that package wget-ssl clashes with wget. Continue with:

cp /opt/bin/wget /opt/bin/wget-old
ipkg remove wget
cp /opt/bin/wget-old /opt/bin/wget
ipkg install libidn
ipkg install optware-devel
 

This time it will finish successfully.

Next we need to install Lame MP3 encoder, providing libmp3lame which will be compiled into FFmpeg, and the Nano text editor (much easier to use than vi), and fix up some other issues (thanks to Nicolas Jolet for these):

ipkg install lame
ipkg install nano
#-----coreutils ls has no colour output by default
mv /opt/bin/ls /opt/bin/ls.bak
ln -sf /bin/ls /opt/bin/ls
#-----coreutils uptime is broken
mv /opt/bin/uptime /opt/bin/uptime.bak
ln -sf /usr/bin/uptime /opt/bin/uptime
 

Download Java SE for Embedded 7 from Oracle, selecting the Linux build for the appropriate CPU. The ARM v5 version is required for the Marvell Kirkwood CPU Synology Products. Note that there is a PowerPC e500v2 version – the CPU core in Synology products which use the Freescale mpc85x3). Unfortunately for PowerPC Synology owners, this depends on a higher version of glibc than the Synology DSM provides for this architecture. Until JamVM supports Java 1.6, or Synology update to glibc 2.4 you won’t be able to follow this guide on PowerPC models. This situation may have changed since DSM 3.2 beta was released.

You will need to sign up to receive the Java download link by email. It’s free to use for non-commercial evaluation use. Use your computer to save it into the top level shared folder of your NAS, which will probably be /volume1/public on the NAS filesystem. Then:

cd /volume1/public
mv ejre-7-fcs-b147-linux-arm-sflt-headless-27_jun_2011.tar.gz /volume1/@tmp
cd /volume1/@tmp
tar xvzf ejre-7-fcs-b147-linux-arm-sflt-headless-27_jun_2011.tar.gz
mv ejre1.7.0 /opt/local/java
 

Synology’s Linux build has no localization support built in, though it does use UTF-8 character encoding for the filesystem. That’s no problem for storage, however the Java VM inherits the locale setting of the host OS. Since this is undefined, Serviio and all other Java software will default to US-ASCII which is a big problem if you have filenames with non-US characters. The solution is to obtain the missing files to add locale support from the Synology toolchain, which is distributed under the GPL. Many thanks to IWarez at the Subsonic forum for this fix, though it took me a good while to realise the required files are indeed included in the ARM toolchain:

#-----for ARM CPU
cd /volume1/@tmp
wget http://sourceforge.net/projects/dsgpl/files/DSM%203.1%20Tool%20Chains/Marvell%2088F628x%20Linux%202.6.32/gcc421_glibc25_88f628x.tgz
tar xvfz gcc421_glibc25_88f628x.tgz
cd arm-none-linux-gnueabi/arm-none-linux-gnueabi/libc/usr/bin
cp locale /opt/bin
cp localedef /opt/bin
cp -R arm-none-linux-gnueabi/arm-none-linux-gnueabi/libc/usr/share/i18n /usr/share
#-----keep another copy safe in case DSM is reinstalled later
cp -R arm-none-linux-gnueabi/arm-none-linux-gnueabi/libc/usr/share/i18n /opt/share
mkdir /usr/lib/locale
localedef -c -f UTF-8 -i en_US en_US.utf8

#-----for Intel CPU
cd /volume1/@tmp
wget http://sourceforge.net/projects/dsgpl/files/DSM%203.1%20Tool%20Chains/Intel%20x86%20Linux%202.6.32/gcc420_glibc236_pineview.tgz
tar xvfz gcc420_glibc236_pineview.tgz
cd i686-linux-gnu/i686-linux-gnu/bin
cp locale /opt/bin
cp localedef /opt/bin
cp -R i686-linux-gnu/i686-linux-gnu/share/i18n /usr/share
#-----keep another copy safe in case DSM is reinstalled later
cp -R i686-linux-gnu/i686-linux-gnu/share/i18n /opt/share
mkdir /usr/lib/locale
localedef -c -f UTF-8 -i en_US en_US.utf8

 

Now use nano to edit some configuration files (Ctrl-o saves, and Ctrl-x exits).
First edit the profile for all bash shell users:

nano /opt/etc/profile
 

Make the changes shown in bold:

#
# Bash initialization script
#

PS1=”[\u@\h \W]$ ”
PATH=/opt/sbin:/opt/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/local/java/bin
LD_LIBRARY_PATH=/opt/lib:${LD_LIBRARY_PATH}
JAVA_HOME=/opt/local/java
LANG=en_US.utf8

export PS1 PATH LD_LIBRARY_PATH JAVA_HOME LANG

 

Save and exit. Next edit the profile for all ash shell users (root):

nano /etc/profile
 

Add the changes shown in bold:

#/etc/profile: system-wide .profile file for ash.
PATH=”$PATH:/bin:/sbin:/usr/bin:/usr/sbin:/usr/syno/bin:/usr/syno/sbin:/usr/local/bin:/usr/local/sbin
PATH=”$PATH:/opt/local/java/bin”
JAVA_HOME=/opt/local/java
LANG=en_US.utf8
export JAVA_HOME LANG

umask 022

 

Save and exit. Now install Serviio 0.6:

cd /volume1/@tmp
wget http://download.serviio.org/releases/serviio-0.6-linux.tar.gz
tar xvzf serviio-0.6-linux.tar.gz
mv serviio-0.6 /opt/local/serviio
 

Next, compile librtmp which FFmpeg will use to retrieve Adobe Flash streaming video, the libssl static libraries (which librtmp depends on), libz and libbz2 static libraries for FFmpeg.

#-----libz static lib (ipkg only provides shared)
wget http://zlib.net/zlib-1.2.5.tar.gz
tar xvfz zlib-1.2.5.tar.gz
cd zlib-1.2.5
./configure --prefix=/opt --static
make
make install
 

At this point ou will see the error “cp: missing destination file operand after `/opt/lib'” because the libz.so shared library wasn’t compiled and the install script tries to copy it nonetheless. libz.a which we need was built ok, so ignore and continue.

cd ..
#-----libbz2 static lib (ipkg only provides shared)
wget http://bzip.org/1.0.6/bzip2-1.0.6.tar.gz
tar xvfz bzip2-1.0.6.tar.gz
cd bzip2-1.0.6
make
make install PREFIX=/opt
cd ..

#-----libssl & libcrypto static libs (ipkg only provides shared)
ipkg install openssl-dev
wget http://www.openssl.org/source/openssl-0.9.8p.tar.gz
tar xvfz openssl-0.9.8p.tar.gz
cd openssl-0.9.8p
./config --prefix=/opt no-shared
make
cp libssl.a /opt/lib
cp libcrypto.a /opt/lib
#-----librtmp won't compile without these pkgconfig definitions
cp libssl.pc /opt/lib/pkgconfig
cp libcrypto.pc /opt/lib/pkgconfig
cd ..

#-----librtmp depends on libpthreads but the optware copy seems to be broken
#-----http://forum.synology.com/enu/viewtopic.php?f=90&t=30132
mkdir /opt/arm-none-linux-gnueabi/lib_disabled
mv /opt/arm-none-linux-gnueabi/lib/libpthread* /opt/arm-none-linux-gnueabi/lib_disabled
cp /lib/libpthread.so.0 /opt/arm-none-linux-gnueabi/lib/
cd /opt/arm-none-linux-gnueabi/lib/
ln -s libpthread.so.0 libpthread.so
ln -s libpthread.so.0 libpthread-2.5.so
cd /volume1/@tmp
wget http://download.serviio.org/opensource/rtmpdump-c58cfb3e9208c6e6bc1aa18f1b1d650d799084e5.tar.gz
tar xvfz rtmpdump-c58cfb3e9208c6e6bc1aa18f1b1d650d799084e5.tar.gz
cd rtmpdump
#-----move all static libs to a separate folder to force compiler to use them
#-----http://www.gossamer-threads.com/lists/apache/dev/265052
mkdir /volume1/@tmp/lib
cp /opt/lib/*.a /volume1/@tmp/lib
#-----fix Makefile (won't compile without libdl linked)
#-----http://forum.luahub.com/index.php?topic=2390.0
sed -i.bak -e '/^LIB_OPENSSL\=/s/lcrypto/lcrypto \-ldl/' Makefile
make SYS=posix prefix=/opt INC=-I/opt/include XLDFLAGS=-L/volume1/@tmp/lib SHARED=
make install prefix=/opt SHARED=
cd ..

#-----gather all static libs again ready for FFmpeg compile
cp /opt/lib/*.a /volume1/@tmp/lib

#-----remove unsupported URL line from /opt/lib/pkgconfig/librtmp.pc
#-----pkg-config --exists --print-errors librtmp
sed -i -e '/^URL/d' /opt/lib/pkgconfig/librtmp.pc

#-----FFmpeg 0.8.2 (slightly newer than the one at Serviio.org)
cd /volume1/@tmp
wget http://dl.dropbox.com/u/1188556/ffmpeg-HEAD-05a2673.tar.gz
tar xvfz ffmpeg-HEAD-05a2673.tar.gz
cd ffmpeg-HEAD-05a2673

#-----for ARM CPU
./configure --arch=arm --enable-armv5te --prefix=/opt --extra-cflags='-I/opt/include' --extra-ldflags='-L/volume1/@tmp/lib' --enable-static --disable-shared --disable-ffplay --disable-ffserver --enable-pthreads --enable-libmp3lame --enable-librtmp --extra-version=Serviio

#-----for Intel CPU
ipkg install yasm
./configure --arch=x86_64 --enable-ssse3 --prefix=/opt --extra-cflags='-I/opt/include' --extra-ldflags='-L/volume1/@tmp/lib' --enable-static --disable-shared --disable-ffplay --disable-ffserver --enable-pthreads --enable-libmp3lame --enable-librtmp --extra-version=Serviio

#-----for PowerPC CPU
./configure --arch=powerpc --disable-altivec --prefix=/opt --extra-cflags='-I/opt/include' --extra-ldflags='-L/volume1/@tmp/lib' --enable-static --disable-shared --disable-ffplay --disable-ffserver --enable-pthreads --enable-libmp3lame --enable-librtmp --extra-version=Serviio

make
make install
 

Notice that the ./configure command is line wrapped – it’s all one command. The make command takes approximately 25 minutes and will show many warnings during compilation, but this is expected.

Try running ffmpeg and check the compile time to make sure the newly compiled one is running. You should see this but with your compilation date and time:

ffmpeg version 0.8.2.git-05a2673-Serviio, Copyright (c) 2000-2011 the FFmpeg developers
built on Aug 22 2011 09:57:40 with gcc 4.2.3
configuration: –arch=arm –enable-armv5te –prefix=/opt –extra-cflags=-I/opt/include –extra-ldflags=’-L/volume1/@tmp/lib’ –enable-static –disable-shared –disable-ffplay –disable-ffserver –enable-pthreads –enable-libmp3lame –enable-librtmp –extra-version=Serviio
libavutil 51. 13. 0 / 51. 13. 0
libavcodec 53. 11. 0 / 53. 11. 0
libavformat 53. 9. 0 / 53. 9. 0
libavdevice 53. 3. 0 / 53. 3. 0
libavfilter 2. 34. 1 / 2. 34. 1
libswscale 2. 0. 0 / 2. 0. 0
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]… {[outfile options] outfile}…

Use -h to get full help or, even better, run ‘man ffmpeg’

In the DSM User Control Panel create a new user called serviio and set a password. Give that user access to the paths that contain the media you want to serve. Click the User Home button and enable the User Home Service. Go back to your SSH session and type:

nano /etc/passwd
 

Be very careful editing this file. A wrong move here could trash your system. Notice that the serviio user has a shell of /sbin/nologin. Change this to /bin/sh like the admin user has. Nano may try to line wrap this line as you type if you added an long account description. If it does, delete the carriage return before the line break and pull it back onto one line. Save and exit.

Now we’ll create the Serviio daemon start and stop script:

nano /volume1/@tmp/S99serviio.sh
 

Paste in the following text (mouseover and use the view source button to copy):

#!/bin/sh

User=serviio

case "$1" in

stop)
        echo "Stop Serviio..."
        su -l $User -c "/opt/local/serviio/bin/serviio.sh -stop" > /dev/null 2>&1 &
        ;;

start)
        # start Serviio in background mode
        su -l $User -c "/opt/local/serviio/bin/serviio.sh" > /dev/null 2>&1 &
        echo "Start Serviio..."
        ;;

restart)
        $0 stop
        sleep 1
        $0 start
        ;;
*)
        echo "usage: $0 { start | stop | restart}" >&2
        exit 1
        ;;

esac
 

Now make it executable and set the serviio user to be the owner of the Serviio folder (so it can create the media database, and log files):

chmod +x /volume1/@tmp/S99serviio.sh
chown -R serviio /opt/local/serviio
 

Test the script manually by running:

/volume1/@tmp/S99serviio.sh start
ps
 

The running process list should show the serviio user running the Serviio launcher shell script and the JVM like so:

 5132 root     16988 S    /usr/syno/sbin/nmbd -D
 5134 admin    34804 S    postgres: admin photo [local] idle
 5187 root     21080 S    /usr/syno/sbin/smbd -D
 5189 root     21080 S    /usr/syno/sbin/smbd -D
 5205 root      7216 S    /usr/syno/sbin/cnid_metad -l log_error
 5207 root     13808 S    /usr/syno/sbin/afpd -g guest -c 256 -n DS111 AFPServ
 5253 root      9208 S N  /usr/syno/sbin/synomkflvd
 5317 root      4112 S    /usr/syno/sbin/sshd
 5332 root      6932 S    /usr/syno/apache/bin/httpd -f /usr/syno/apache/conf/
 5344 root      7068 S    /usr/syno/apache/bin/httpd -f /usr/syno/apache/conf/
 5345 root      7068 S    /usr/syno/apache/bin/httpd -f /usr/syno/apache/conf/
 5369 root      6928 S    /usr/syno/apache/bin/httpd
 5383 nobody    6928 S    /usr/syno/apache/bin/httpd
 5384 nobody    6928 S    /usr/syno/apache/bin/httpd
 5385 nobody    6928 S    /usr/syno/apache/bin/httpd
 5436 root      2408 S    avahi-daemon: running [DS111.local]
 5460 root     11928 S    sshd: root@pts/0
 5466 root      5452 S    -ash
 5582 root         0 SW   [flush-8:0]
 5602 serviio   2988 S    -sh -c /opt/local/serviio/bin/serviio.sh
 5605 serviio   2988 S    /bin/sh /opt/local/serviio/bin/serviio.sh
 5610 serviio   498m S    /opt/local/java/bin/java -Xmx384M -Djava.net.preferI
 5623 root      5452 R    ps
 

Stop the daemon with:

/volume1/@tmp/S99serviio.sh stop
 

Check the web UI or the process list again and make sure it did indeed stop. If it’s all ok, we need to move the daemon launcher script so it starts automatically on boot (it will survive a DSM upgrade):

mv /volume1/@tmp/S99serviio.sh /opt/etc/init.d
 

Shutdown the NAS and restart, checking that Serviio starts by itself.

From now on you can start and stop Serviio manually using this same script (say if you were experimenting with changes to profiles.xml for instance):

/opt/etc/init.d/S99serviio.sh stop
/opt/etc/init.d/S99serviio.sh start
 

Remote console

AcidumIrae’s PHP Web UI is only for Serviio up to version 0.5.2 at present, and at the time of writing the Java Restful Web UI by kairoh does not yet integrate with Serviio 0.6. The only option for now is to use the proper Serviio Console from another computer. Since 0.6 there is no longer a requirement to change anything at the server end, so you just need to pass the parameter -Dserviio.remoteHost=x.x.x.x to the Serviio Console to indicate the IP of your Serviio server. This is covered in the Serviio FAQ. There’s quite a lag for it to start up (at least on Windows), but eventually you see the familiar icon.

 

Serviio settings

Navigate to the Transcoding tab. Set the transcoded files location to /volume1/@tmp. Failure to do this will result in temporary files being written to /tmp which will fill up the partition it’s on the moment you remux a 4GB movie, which would prevent you from logging into the NAS. If your Synology has an Intel CPU set the number of transcoding threads to 2. The dual cores should take advantage of FFmpeg’s pthreads support for a decent performance boost.

I recommend de-selecting Generate thumbnails for local videos in the Metadata tab. Note that you have to click Save on each tab or any change will not take effect. As you’ll see if you have a DLNA renderer that supports thumbnails, Serviio retrieves good ones from the online databases it checks so they’re not really needed. Often FFmpeg will get stuck trying to generate a thumbnail for a video and will lock the CPU at 100% for long periods of time. This issue was raised in this Serviio forum thread.

You can use the Library tab to add the media folders you want to share. Use Add Path not Add Local (we’re using a remote console remember) and express the paths in Linux syntax, e.g. /volume1/public/videos. Be sure that the serviio user has been granted read privileges over the folders you add (User Control Panel in DSM).

 

Performance

The Synology seems perfectly able to transcode DTS audio in a hi-def Matroska file down to 2 channel AC3 in MPEG-TS while copying the H.264 stream. The CPU use leaps up to 100% but I guess that’s because it’s running ahead transcoding down to the end of the file (and is CPU-bound).

If I play something that’s only remuxing the container and copying both audio and video streams then the CPU stays at around 40% (because it’s I/O-bound) then 5 minutes into a film it falls away to pretty much idle – I guess it has finished remuxing to the temporary file.

Memory use even while running Serviio is around 20% at idle, 30% during a remux with audio transcode, though I have noticed that with a larger library this creeps up to around 70%. I have a 2011 product so it has 256MB of RAM. This seems to indicate that the value line of products (with the j suffix) like the DS211j should run Serviio on their 128MB. They have a 1.2GHz CPU though versus 1.6GHz on mine (and the DS110j is only 800MHz), so it would need testing by someone.

All in all it would seem that the Synology products are very capable Serviio appliances!

 

Uninstalling

Log in as root and stop Serviio, then in the DSM User Control Panel delete the user called serviio. Undo the changes that you made to /opt/etc/profile and /etc/profile (the LANG, JAVA_HOME and java path modifications marked in bold earlier in the guide).

Finally, re-download the bootstrap for your model of NAS and run it again. It will tell you to delete a few folders and restart. This will completely trash all optware ipkg packages (i.e. everything in /opt) and undo everything else you did in this guide without affecting your data partitions. If you had any other ipkg packages installed since installing Serviio, these would also be lost.

Cisco wifi WPA2-Enterprise PEAP authentication with Active Directory

Business users increasingly expect full LAN access while working wirelessly around the workplace. On account of the perceived weakness of WPA cryptography many network administrators will tend to offer a separate guest network over wifi, but not the full corporate LAN. I was one of those people until I got this working six months ago. This solution supports Mac, PC clients, together with iOS devices (iPhone, iPad), and I would guess Linux too since it’s based on open standards.

There is a pretty comprehensive Cisco configuration example document on this subject, but there isn’t much information on the web apart from that. There are several drawbacks to that guide:-

  • it only outlines the GUI config of a centralised Cisco Wireless LAN Controller (not standalone Cisco wifi access points)
  • it deals with Windows Server 2003 IAS rather than its more current replacement: NPS, as introduced by Windows Server 2008.
  • it assumes that the client will be running Cisco wifi client software (presumably it pre-dates the introduction of Windows’ own Wireless Network control panel introduced with XP Service Pack 1).

The access points authenticate using PEAP which is EAP inside a TLS session, therefore a working PKI is required for creating the NPS server’s certificate. If you don’t already have it, install the Active Directory Certificate Services role to one of your Windows domain servers. As I mentioned in my LDAPS guide, that whole process is somewhat outside the scope of this blog post but do heed Microsoft’s warning:

Warning Before you install a certification authority (CA), you should be aware that you are creating or extending a public key infrastructure (PKI). Be sure to design a PKI that is appropriate for your organization. See PKI Design Brief Overview for additional information.

 

Windows PKI Problems

DCs should auto-enroll for their own certificates once that’s up and running. However I had huge difficulties with this. I installed the Active Directory Certificate Services role on a Windows Server 2008 R2 Domain Controller. I soon discovered that none of the other 2008 R2 DCs could auto-enroll for certificates (the Event Log reported RPC server unavailable in the failure event which was quite misleading). I could not manually enroll for certificates using the Certificates MMC snap-in either.

This held me up for a long time, and I was able to find several sources stating that the solution was to add the Domain Controllers to the group CERTSVC_DCOM_ACCESS, which didn’t work for me. In the end I found the solution in this Technet answer by Joson Zhou. Somebody in the distant past had apparently tampered the Active Directory group membership of Builtin\Users (CN=Users,CN=Builtin,DC=domain,DC=com) and had removed Interactive and Authenticated Users. It had been like this for years with no adverse impact. Since a DC has no local accounts it will use this AD group as its definition of which users have User privileges. I suppose that up until this time no service that would have been affected had been installed on a DC.
I called a business contact to verify against an external Active Directory and, sure enough, those missing groups should indeed have been members. The moment I corrected this I was able to enroll for a DC certificate. Sadly the Technet thread is locked so I wasn’t able to thank Joson, and unfortunately it seems you can’t send private messages. I don’t understand why anyone would be inclined to interfere with default Active Directory groups – searching only reveals this post. The guy asks if it’s best practice to remove Authenticated Users from that group and is pretty roundly slapped down. Perhaps people did this with NT 4.0, which my domain dates from.

I also had a recurring issue in the Event Logs where these DCs would fail to pull down the Certificate Revocation Lists from Microsoft and fail to refresh the Trusted Root certificates. Microsoft KB931125 was the only thing that fixed that. However, while troubleshooting this problem I ended up installing the Trusted Root CA update pack (from here I think), assuming I’d be able to uninstall it later if it didn’t improve things. Unfortunately it cannot be uninstalled, and it caused me another huge hold-up later on.

 

Configuring Network Policy Server

Once your Domain Controller has a certificate you can install the Network Policy Server role. Each access point (or only your WLC if you use LWAPP access points) will need to be on a static IP address, and have an entry in the RADIUS clients section of the NPS management MMC. When generating Pre-Shared Keys note that it seems Cisco devices do not tolerate keys containing a dollar sign, at least not in their WebUI.

NPS RADIUS client settings

The behaviour for Windows computers is that before logon the PC will authenticate and connect using its computer account (providing connectivity to the domain), then when the user logs in the wifi network is re-connected in that user context. You will therefore probably need an NPS Network Policy item for both cases (NPS -> Policies -> Network Policies) which will allow you to apply different settings to each (timeouts, etc.):

NPS policy for wireless computers
NPS policy for wireless users

In each of the policies be sure to select PEAP as the only EAP authentication type, with EAP-MSCHAP v2 as below, selecting your server’s certificate in the drop-down.
NPS server PEAP encryption settings

Make sure that your Connection Request Policies are not preventing connections (NPS -> Policies -> Connection Request Policies) – I think they are disabled by default.

 

Access Point Configuration

I won’t bother to describe the Cisco WLC configuration, since that is basically identical to Cisco’s own guide. However, one problem I had in testing was that I removed and re-added the RADIUS server settings on the WLC and I had forgotten that you need to specify your NPS server twice – once for Authentication, and separately once again for Accounting (assuming you’re using accounting), which is not the case in the Server Manager GUI on standalone access points:

Wireless LAN Controller RADIUS server settings

Not everyone has the budget for a WLC, or you might, like me, have combination of a WLC at one site but not at others. It’s not practical to use a single WLC for multiple sites since all LWAPP traffic is trunked back to the controller – you wouldn’t want that load on your site-to-site links. Here is a sample multiple SSID configuration for a standalone Cisco 1131AG Access Point (AIR-AP1131AG-E-K9). I’m not going to provide Cisco access point GUI screenshots, but if you want them the first part of the following guide covers that well:
http://blog.laurence.id.au/2010/03/running-peap-with-cisco-aeronet-1231g.html

My example access point config below relates to the following schematic. If you’re going to apply this config to a bare unit, substitute the IPs and VLANs to match your environment and remove the lines with keys or passwords.

Sample access point config schematic

!
! Last configuration change at 17:13:17 UTC Tue Feb 22 2011 by admin
! NVRAM config last updated at 17:14:03 UTC Tue Feb 22 2011 by admin
!
version 12.4
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname test-ap1
!
logging rate-limit console 9
enable secret 5 3242576EABCD3F3254$2
!
aaa new-model
!
!
aaa group server radius rad_eap
 server 172.16.2.50 auth-port 1645 acct-port 1646
!
aaa group server radius rad_mac
!
aaa group server radius rad_acct
 server 172.16.2.50 auth-port 1645 acct-port 1646
!
aaa group server radius rad_admin
!
aaa group server tacacs+ tac_admin
!
aaa group server radius rad_pmip
!
aaa group server radius dummy
!
aaa authentication login eap_methods group rad_eap
aaa authentication login mac_methods local
aaa authorization exec default local
aaa accounting network acct_methods start-stop group rad_acct
!
aaa session-id common
ip domain name domain.com
ip name-server 208.67.222.222
ip name-server 208.67.220.220
!
!
dot11 syslog
dot11 vlan-name Wifi-Corp-WLAN vlan 20
dot11 vlan-name Wifi-Guest-WLAN vlan 6
!
dot11 ssid Corp-WLAN
   vlan 20
   authentication open eap eap_methods
   authentication key-management wpa version 2
   accounting acct_methods
   mbssid guest-mode
!
dot11 ssid Guest-WLAN
   vlan 6
   authentication open
   authentication key-management wpa
   guest-mode
   mbssid guest-mode
   wpa-psk ascii 7 123475652142BCD122
!
power inline negotiation prestandard source
!
!
username admin privilege 15 secret 5 131412ABDEFEE42323
!
!
bridge irb
!
!
interface Dot11Radio0
 no ip address
 no ip route-cache
 !
 encryption mode ciphers tkip
 !
 encryption vlan 20 mode ciphers aes-ccm
 !
 encryption vlan 6 mode ciphers aes-ccm tkip
 !
 ssid Corp-WLAN
 !
 ssid Guest-WLAN
 !
 mbssid
 station-role root
!
interface Dot11Radio0.20
 encapsulation dot1Q 20 native
 no ip route-cache
 bridge-group 1
 bridge-group 1 subscriber-loop-control
 bridge-group 1 block-unknown-source
 no bridge-group 1 source-learning
 no bridge-group 1 unicast-flooding
 bridge-group 1 spanning-disabled
!
interface Dot11Radio0.6
 encapsulation dot1Q 6
 no ip route-cache
 bridge-group 6
 bridge-group 6 subscriber-loop-control
 bridge-group 6 block-unknown-source
 no bridge-group 6 source-learning
 no bridge-group 6 unicast-flooding
 bridge-group 6 spanning-disabled
!
interface Dot11Radio1
 no ip address
 no ip route-cache
 !
 encryption mode ciphers tkip
 !
 encryption vlan 20 mode ciphers aes-ccm
 !
 encryption vlan 6 mode ciphers aes-ccm tkip
 !
 ssid Corp-WLAN
 !
 ssid Guest-WLAN
 !
 no dfs band block
 mbssid
 channel dfs
 station-role root
!
interface Dot11Radio1.20
 encapsulation dot1Q 20 native
 no ip route-cache
 bridge-group 1
 bridge-group 1 subscriber-loop-control
 bridge-group 1 block-unknown-source
 no bridge-group 1 source-learning
 no bridge-group 1 unicast-flooding
 bridge-group 1 spanning-disabled
!
interface Dot11Radio1.6
 encapsulation dot1Q 6
 no ip route-cache
 bridge-group 6
 bridge-group 6 subscriber-loop-control
 bridge-group 6 block-unknown-source
 no bridge-group 6 source-learning
 no bridge-group 6 unicast-flooding
 bridge-group 6 spanning-disabled
!
interface FastEthernet0
 no ip address
 no ip route-cache
 duplex auto
 speed auto
!
interface FastEthernet0.20
 encapsulation dot1Q 20 native
 no ip route-cache
 bridge-group 1
 no bridge-group 1 source-learning
 bridge-group 1 spanning-disabled
!
interface FastEthernet0.6
 encapsulation dot1Q 6
 no ip route-cache
 bridge-group 6
 no bridge-group 6 source-learning
 bridge-group 6 spanning-disabled
!
interface BVI1
 ip address 172.16.2.238 255.255.255.0
 no ip route-cache
!
ip default-gateway 172.16.2.254
ip http server
no ip http secure-server
ip http help-path http://www.cisco.com/warp/public/779/smbiz/prodconfig/help/eag
ip radius source-interface BVI1
snmp-server community public RO
radius-server attribute 32 include-in-access-req format %h
radius-server host 172.16.2.50 auth-port 1645 acct-port 1646 key 7 14154A12D56B57231FDC235331292321631B212212034332
radius-server vsa send accounting
bridge 1 route ip
!
!
!
line con 0
line vty 0 4
!
sntp server 130.159.196.118
sntp server 195.66.148.150
sntp server 84.45.87.84
sntp broadcast client
end
 
 

The Final PKI Hurdle

Each time I tried to authenticate I got an Schannel Event Log error on the NPS server of “The message received was unexpected or badly formatted.” which exactly matched the symptoms described in Microsoft KB933430, although that was only intended for Windows Server 2003. This was confusing, but according to that article:

When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificate authorities trusted for client authentication and remove those that do not really need to be trusted.

I had been working on this issue sporadically and I had only got as far as getting the Enterprise CA online before going on holiday. Only later did I make the connection and remember the updated Trusted Root CA pack that I had loaded on in desperation. Consulting the Certificates MMC snap-in I discovered that the server had 304 trusted root CAs instead of nine! Windows Server 2008 and 2008 R2 do have a more generous storage allowance for sending CA certificates in the PEAP handshake but clearly 304 certificates was too much. Using another server as a reference machine I manually deleted all the superfluous CA certificates and I could finally authenticate via wifi!

 

Non-Domain Client Configuration

When connecting from non-domain machines or iOS devices, the issuing CA for the NPS server’s certificate will not be trusted. In iOS you will be prompted to accept it manually, but the situation is more complicated in Windows 7 (I haven’t tested older OS versions). In my tests the connection would be rejected upon providing the additional user credentials. You will need to export the CA server certificate by running the following command on your CA:

certutil -ca.cert c:\temp\domain-SERVERNAME-CA.cer
 

On the Windows 7 client use the Certificates MMC snap-in to import it into the user’s Trusted Root CA Certificate Store (run MMC.EXE, File -> Add/Remove Snap-in -> Certificates -> My User Account -> Trusted Root Certification Authorities -> Certificates -> right-click -> All Tasks -> Import).

Now if you try to connect you will get the following message:

The Credentials provided by the server could not be validated. We recommend that you terminate the connection and contact your administrator with the information provided in the details. You may still connect but doing so exposes you to a security risk by a possible rogue server.

 

I found that I couldn’t connect even if I ignored this warning. To get around this, you will need to create the wireless network manually using the following settings:

Manual connection settings for Windows 7 (part 1)

Make sure to manually select your CA which should be listed as below:

Manual connection settings for Windows 7 (part 2)

 

Windows AD Domain Client Configuration by Group Policy

Domain workstations can have their wireless networking configs entirely managed by Group Policy, including network preference order, auto connect options, whether to cache credentials and more. Domain members also implicitly trust the Domain CA. Be aware that Windows XP workstations are configured by a separate policy to Windows Vista and Windows 7. Both Group Policies are located at Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Wireless Network (IEEE 802.11) Policies. It’s all pretty self-explanatory, and the security and encryption settings are broadly identical to those in the screenshots above, except that you do not need to specify the CA.

Remember to add your designated wifi users and computers to the AD groups you created for the NPS Network Policies!

iPad, iPhone, and Mac OS X L2TP/IPsec VPN to Windows Server 2008 R2

I spent quite a while experimenting with L2TP over IPsec with my iPad 2, and surprisingly found no useful guides as to how to configure it. Judging by what I could find online, most people simply give up and use PPTP instead which has significant security vulnerabilities. Here’s a concise comparison of PPTP versus L2TP/IPsec which describes that weakness:
http://www.ivpn.net/pptp-vs-l2tp-vs-openvpn.php

I had considered using Apple’s support for Cisco IPsec but that would have meant exposing the core switch where I work. It’s old enough to make that a bad idea. The Juniper Netscreen firewall only supports L2TP with certificates and not Pre-Shared Key so that was also ruled out. This post will outline how to configure Windows Server 2008 R2’s NPS/RRAS role to host L2TP/IPsec connections which will allow iPads and iPhones to connect securely into your Windows infrastructure without the need for additional client software.

Firstly, it’s likely that your NPS/RRAS server is behind a perimeter firewall. If this is the case you’ll need to grant IPsec traffic access from the public internet. Using details from this Technet post I created the following custom service object on the Netscreen firewall, and allowed it inbound to the RRAS server (IP protocols 50 and 51, UDP 500 and 4500). For initial testing though you should probably create a rule to allow all traffic to and from your test client.

IPSec service definition

I am going to assume a knowledge of both NPS and RRAS. For more information on those, other guides exist. As far as I have been able to discover, it seems that the iPad only supports Pre-Shared Key authentication for the IPsec tunnel, rather than certificates-based. The VPN connection settings GUI in Mac OS 10.6 for instance will allow either method, but not in iOS. It may be possible to force your way around this with the iPhone Configuration Utility (designed for applying corporate settings to iOS) but information is pretty scant. I did find a long forum thread about certificate auto-enrollment, and a Microsoft Directory Services team blog post, but I suspect they may relate more to 802.1x:
https://discussions.apple.com/message/10402090
http://blogs.technet.com/b/askds/archive/2010/11/22/ipad-iphone-certificate-issuance.aspx

The L2TP/IPsec Pre-Shared Key is configured by right-clicking on the top level of Routing and Remote Access in Server Manager -> Properties -> Security tab:
Pre-Shared Key for IPsec tunnel

It’s useful to keep your VPN clients on a different subnet to your servers, however multihoming with several NICs can cause problems, particularly if your RRAS server is also a Domain Controller. You can define a subnet for this purpose in the IPv4 tab here, but you will need to remember to add a static route entry on your router pointing traffic for this subnet to the RRAS server.

RRAS client subnet settings

In Server Manager -> NPS -> Policies -> Network Policies create a policy with the following settings, making sure to set the encryption settings. As this Microsoft KB article makes clear, these options actually ensure that IPsec gets used, with the different grades here representing different algorithm proposal combinations. The iPad supports the maximum encryption setting.

NPS Policy Settings for L2TP/IPSec

Lastly, the Mac OS X and iOS VPN client configuration is pretty self-explanatory. Make sure to use the Pre-Shared Key that you defined on the RRAS server (referred to here as Secret):

iPad L2TP VPN configuration

I would at this point like to thoroughly recommend iTap RDP as being the best iOS Remote Desktop client I have seen. It has NLA authentication support, a universal iPad/iPhone binary, and by far the most intuitive controls which really puts it ahead of the competition.

UPDATE – I was hoping to use this VPN configuration for all clients, but it seems that Mac OS clients cannot connect. Mac OS apparently didn’t use the standard L2TP UDP port 1701. Someone compiled a fix for Snow Leopard but I could not get it to work. It’s possible that this is all out of date information though.

UPDATE 2 – I did some more troubleshooting from home and discovered that when a tunnel is initiated from a second device on my home network while another tunnel is already up, all further connection attempts then fail for a long while, even when the RRAS server is rebooted. This would suggest that the Netscreen firewall at my work still considers the original session open, and thus it will eventually timeout after 30 minutes. This behaviour had disrupted my Mac OS X test results. Using verbose logging on the Mac and looking at the NPS log I could see that Mac OS X 10.6.8 VPN client does not accept the 128bit encryption setting. Permitting 56bit encryption allows Macs to connect, but perhaps older versions of Mac OS could have difficulties. I have updated the policy settings screenshot above.

UPDATE 3 – I realised that although NATed clients could connect, clients with public addresses could not. I have amended the destination ports for IP protocols 50 and 51 in the firewall IPsec definition screenshot (it had defaulted to 0-0 rather than 0-65535 for some reason). I have verified that this VPN works for Windows XP clients, Windows 7, Mac OS X 10.6, and Mac OS X 10.5, as well as iPhones (mine’s on iOS 3.1.3) and iPads. Once connected to the RRAS server you cannot interact with that server directly, so make sure that the RRAS server’s own DNS settings do not refer to itself as a primary (assuming it’s also a DNS server) – these DNS entries will be inherited by all VPN clients.

Secure Active Directory authentication for non-domain DMZ web sites using LDAPS

Being able to have people log into websites using credentials they already use is a huge advantage. Hosting those websites in DMZ zones away from domain controllers means that you need to do this via LDAP, but this isn’t secure since the query results are sent in plain text. Enter LDAPS, which from the research I did online seems to be the most misunderstood technology I have used to date, with very few documented examples.

It’s a pretty simple idea – wrap LDAP in a TLS session. The LDAP client connects to the server. The server sends over its certificate, which is used to authenticate it as being the actual server the client expects. This is critical since the webserver is delegating authentication to it. A TLS session is established, and the client binds to the directory with credentials sent in plain text, but TLS provides the security against eavesdropping.

What is absolutely baffling is how many big companies are completely oblivious to getting this working correctly. I deal with one of the big players in the email scanning space which offers LDAPS authentication so customers can use their normal Active Directory credentials to check their spam. I had bought a £500 VeriSign certificate expressly for this purpose, but when this certificate expired I discovered that this company still had connectivity to the domain controller. They are clearly not even checking the chain of trust on the certificate, even though they allow it to authenticate users! It would be relatively trivial to exploit this to impersonate a particular domain controller, insert an appropriate dummy user account, and hack any of their customers who use federated authentication. Security by DNS is not security (think man-in-the-middle, or cache-poisoning).

I have also worked with a supposedly major website CMS system which cannot get this right, despite claims by the vendor that it supports LDAPS. It categorically doesn’t – LDAP on a different TCP port is still LDAP.

Concept summary

There seems to be a lot of conflicting information out there on how this works and how the certificates are involved, so this is the overview as I understand it:

  • The LDAPS client initiates a connection to the DC on TCP636 specifiying the DC’s FQDN (IP address or hostname only are not sufficient).
  • The DC checks through its Local Computer certificate store for a certificate with the same FQDN, then signs using its private key as proof of ID. The certificate is sent to the client, so the client gets the DC’s public key.
  • The LDAPS client checks the chain of trust on that certificate. If the CA is not a trusted public one (like VeriSign) the client system will need to have the issuing CA’s own certificate pre-loaded into its Local Computer Trusted Root Certification Authorities certificate store. Remember, the web server is not in the Active Directory domain so it won’t implicity trust that CA.
  • Once the client trusts that the DC is genuine, the SSL session is established and the client can bind to the directory using plain text Active Directory account credentials.
 

Configuring your Domain Controller for LDAPS

This knowledgebase article outlines the steps the DC goes through to select a certificate, and includes details of how to create the CSR for external certification authorites: http://support.microsoft.com/kb/321051

The information there is pretty good, and there aren’t many gotchas. If you’re aiming for third party servers to bind to your AD then you will probably want an SSL certificate from a commercial CA. Be aware that you will need a certificate with a Subject Alternate Name (to have two hostnames registered) if you use a different internal FQDN for that server. The cheap public CAs like GoDaddy could not offer this the last time I enquired (admittedly about two years ago). If the website that will be binding to your directory is your own then you won’t want to buy a certificate. If you don’t already have it, install the Active Directory Certificate Services role to one of your Windows domain servers. That whole process is somewhat outside the scope of this blog post. Heed Microsoft’s warning from their Technet LDAP over SSL wiki:

Warning Before you install a certification authority (CA), you should be aware that you are creating or extending a public key infrastructure (PKI). Be sure to design a PKI that is appropriate for your organization. See PKI Design Brief Overview for additional information.

DCs should auto-enroll for their own certificates once that’s up and running. As per that Microsoft KB article that’s all you need to do. DCs will respond to LDAPS requests providing they can find a valid SSL certificate. I did run into some issues setting up my PKI which I have documented in another post.

If you’re using your own Active Directory domain’s CA, export the CA’s own certificate by running:

certutil -ca.cert c:\temp\domain-SERVERNAME-CA.cer
 

Configuring the DMZ webserver

Assuming your domain controller isn’t in public DNS, add its FQDN to the hosts file on your DMZ webserver.

Use the Certificates MMC snap-in to import the CA’s certificate you exported earlier (run MMC.EXE, File -> Add/Remove Snap-in -> Certificates -> Computer Account -> Local Computer -> Trusted Root Certification Authorities -> Certificates -> right-click -> All Tasks -> Import).

Example C# code snippet for the website:

NetworkCredential networkCredentials = new System.Net.NetworkCredential(adminUsername, adminPassword);

LdapDirectoryIdentifier ldapDirectoryIdentifier = new LdapDirectoryIdentifier(ldapServer, ldapPort);
LdapConnection ldapConnection = new LdapConnection(ldapDirectoryIdentifier, networkCredentials, AuthType.Basic);

ldapConnection.SessionOptions.SecureSocketLayer = true;
ldapConnection.Bind();
 

Verifying your LDAPS server using OpenLDAP

Since in my case I was not developing the .Net code myself, I found it useful to use OpenLDAP to demonstrate to our developers that the server and firewall were definitely working as expected. I had been unable to get the normal Windows LDAP troubleshooting tool LDP.EXE to work and it’s not particularly verbose during failures (as it turns out, I hadn’t been using the FQDN). OpenLDAP is extremely easy to install on any Linux distro with a package management system. I ended up using my Synology NAS at home. The format of the issuing CA certificate needs to be slightly different though (base64 encoded). On the CA server, use:

certutil -ca.cert c:\temp\domain-SERVERNAME-CA.cer
certutil -encode c:\temp\domain-SERVERNAME-CA.cer c:\temp\domain-SERVERNAME-CA.pem
 

Add the following line to OpenLDAP’s ldap.conf (mine was /opt/etc/openldap/ldap.conf):

TLS_CACERT /opt/etc/openldap/domain-SERVERNAME-CA.pem
 

Make sure of course to copy the .pem file to that location.
The ldapsearch syntax is quite a minefield but my sample query was as follows:

ldapsearch -x -v -H 'ldaps://ldap.domain.com' -b 'dc=domain,dc=com' -s base -D 'user@domain.com' -W
 

which binds to the directory as user@domain.com (you are prompted for a password), and requests only the top level directory objects.

It seems to be very difficult to compile OpenLDAP for Windows, in fact it’s actually impossible using MinGW (one of the dependencies CyrusSASL needs Visual Studio to compile).

I tried to use this binary build of OpenLDAP for Windows but it ignores ldap.conf because the people who compiled it forgot to define the variable sysconfdir in the configure script. I was able to use the excellent Process Monitor to track a failed filesystem read to a path featuring the static entry $SYSCONFDIR$ as ldapsearch.exe is invoked. Unfortunately as such you can’t add the CA cert location to the config.

Batch script for recursive FFmpeg transcoding

I recently had to shrink around 50GB of MP3 audio recordings that were sitting in a nested folder structure on a web server. Having experimented to find more appropriate LAME encoder settings for spoken word content I needed to transcode the files whilst keeping the existing ID3 tags intact. FFmpeg can do this nicely using libmp3lame, whereas LAME by itself cannot. Armed with my own compile of FFmpeg, I created a drag & drop batch script to recursively work through a folder structure transcoding MP3 and WAV files and writing out the resulting MP3 files in the same folder structure with an amended top level folder name. It will accept multiple files or folders being dragged and dropped. You could adapt this script to whatever task you’re using FFmpeg for.

::MP3 transcoding to more sensible default quality settings for spoken word
::32kHz mono VBR @ approx. 64Kbps
::
::pcloadletter.co.uk

@echo off
setlocal ENABLEDELAYEDEXPANSION
::parse multiple command lines (so multiple targets can be dragged and dropped at once)
:commandlineloop
if "%~1"=="" goto :continue
call :mainloop %1
shift
goto :commandlineloop
:continue
echo.
pause
goto :eof

::main loop we run for each command line arg of the script
:mainloop
set folderpath=
if "%~1"=="" (
  echo Drag and drop files or folders onto this script.
  goto :eof
)
set folderpath=%~1
echo "%1" | find /I ".mp3" && (
  goto :fileonly
)
echo "%1" | find /I ".wav" && (
  goto :fileonly
)

::folders
echo.
echo.
echo Processing folder %1
echo ________________________________________________________________________________
echo.

::we need to find the last folder level in the folderpath
::batch scripting is horrible so we need to use a hacky way to get the last token
::(from http://stackoverflow.com/questions/5473840/last-token-in-batch-variable)
set temp_string=%~1
set count=0

::iterate parsing the tokens and trimming the string until it's empty, while counting how many loops
:loopcounter
for /F "tokens=1* delims=\" %%a in ( "%temp_string%" ) do (
  set /A count+=1
  set temp_string=%%b
  goto loopcounter
)
for /F "tokens=%count% delims=\" %%i in ( "%~1" ) do set folder=%%i

::append the top level folder name with "-optimized" so we create a new folder tree
for /R "%folderpath%" %%i in (*.mp3 *.wav) do (
  set sourcepath=%%~di%%~pi
  set destpath=!sourcepath:%folder%=%folder%-optimized!
  md "!destpath!"
  ffmpeg -i "%%~fi" -acodec libmp3lame -aq 8 -ar 32000 -ac 1 "!destpath!%%~ni.mp3"
)
goto :eof

::individual files
:fileonly
for %%i in ("%folderpath%") do (
  ffmpeg -i "%folderpath%" -acodec libmp3lame -aq 8 -ar 32000 -ac 1 "%%~di%%~pi%%~ni-optimized.mp3"
)
 

There are several neat little tricks here. Multiple folders can be dragged onto the script because it uses SHIFT to work through each command line argument. The next hurdle is to get the last part of the folder name(s) being dragged. Though this would be simple in any Unix-like shell, but to do this in batch without relying on any additional tools proved to be quite tricky. Though tokens can be parsed by FOR, finding the last one (which we need) requires us to count how many tokens there are. Only then can the last one be selected. The folder name string substitution is done by the SET command (also used in my file renaming script). Delayed variable expansion means that the variables between the the exclamation marks are evaluated once per loop rather than the default of once per script execution (more info here). I hadn’t realised until today that you can create a folder structure several layers deep with a single MD command. This avoids having to iterate through all subfolders – we can use FOR’s /R switch to handle the recursion in a single line. For more information on some of the variables containing tildes like %%~pi and %%~ni, try running FOR /?. The script also first runs the FFmpeg subroutine with ECHO in front of the commands so you can double-check the syntax before proceeding.

There are of course much more efficient ways of compressing spoken word content than MP3 these days – AAC-HEv2 for instance, but that will rule out all but the latest audio playback devices.
The source MP3 recordings were in 44.1kHz mono @ 128Kbps. This was definitely overkill for speech. Just by opting for 64Kbps I could halve that. However I couldn’t drop the sample rate below 32kHz, as this is the lowest legal limit of MPEG-1 layer III. This info is tricky to find but the LAME encoder reveals it in its extended help:

MPEG-1   layer III sample frequencies (kHz):  32  48  44.1
bitrates (kbps): 32 40 48 56 64 80 96 112 128 160 192 224 256 320
MPEG-2   layer III sample frequencies (kHz):  16  24  22.05
bitrates (kbps):  8 16 24 32 40 48 56 64 80 96 112 128 144 160
MPEG-2.5 layer III sample frequencies (kHz):   8  12  11.025
bitrates (kbps):  8 16 24 32 40 48 56 64

 

I think it’s safe to assume that most MP3 players can handle variable bit rate. As far as I can remember only the first generation ones couldnt from early last decade – the sort of ones which only had around 64MB of storage. I don’t think excluding players this old is much of a concern.

Playing around with VBR quality settings in LAME we can see that with -V 9 we get a small file (12.9MB becomes 3.7MB) however it’s actually an MPEG-2 layer III file which as you can see above supports lower sample rates. I’m not certain of the wider compatibility of this sub-type of MP3 file, and besides it does sound considerably worse than the lowest MPEG-1 layer III settings which is –V 8 (12.9MB becomes 5.2MB). This proved to be the best compromise between size and quality.