Category Archives: Scripting

CrashPlan packages for Synology NAS

UPDATE – The instructions and notes on this page apply to all three versions of the package hosted on my repo: CrashPlan, CrashPlan PRO, and CrashPlan PROe.

CrashPlan is a popular online backup solution which supports continuous syncing. With this your NAS can become even more resilient – it could even get stolen or destroyed and you would still have your data. Whilst you can pay a small monthly charge for a storage allocation in the Cloud, one neat feature CrashPlan offers is for individuals to collaboratively backup their important data to each other – for free! You could install CrashPlan on your laptop and have it continuously protecting your documents to your NAS, even whilst away from home.

CrashPlan-Windows

CrashPlan is a Java application, and one that’s typically difficult to install on a NAS – therefore an obvious candidate for me to simplify into a package, given that I’ve made a few others. I tried and failed a few months ago, getting stuck at compiling the Jtux library for ARM CPUs (the Oracle Java for Embedded doesn’t come with any headers).

I noticed a few CrashPlan setup guides linking to my Java package, and decided to try again based on these: Kenneth Larsen’s blog post, the Vincesoft blog article for installing on ARM processor Iomega NAS units, and this handy PDF document which is a digest of all of them, complete with download links for the additional compiled ARM libraries. I used the PowerPC binaries Christophe had compiled on his chreggy.fr blog, so thanks go to him. I wanted make sure the package didn’t require the NAS to be bootstrapped, so I picked out the few generic binaries that were needed (bash, nice and cpio) directly from the Optware repo.

UPDATE – For version 3.2 I also had to identify and then figure out how to compile Tim Macinta’s fast MD5 library, to fix the supplied libmd5.so on ARM systems (CrashPlan only distributes libraries for x86). I’m documenting that process here in case more libs are required in future versions. I identified it from the error message in log/engine_error.log and by running objdump -x libmd5.so. I could see that the same Java_com_twmacinta_util_MD5_Transform_1native function mentioned in the error was present in the x86 lib but not in my compiled libmd5.so from W3C Libwww. I took the headers from an install of OpenJDK on a regular Ubuntu desktop. I then used the Linux x86 source from the download bundle on Tim’s website – the closest match – and compiled it directly on the syno using the command line from a comment in another version of that source:
gcc -O3 -shared -I/tmp/jdk_headers/include /tmp/fast-md5/src/lib/arch/linux_x86/MD5.c -o libmd5.so

Aside from the challenges of getting the library dependencies fixed for ARM and QorIQ PowerPC systems, there was also the matter of compliance – Code 42 Software’s EULA prohibits redistribution of their work. I had to make the syno package download CrashPlan for Linux (after the end user agrees their EULA), then I had to write my own script to extract this archive and mimic their installer, since their installer is interactive. It took a lot of slow testing, but I managed it!

CPPROe package info

My most recent package version introduces handling of the automatic updates which Code 42 sometimes publish to the clients. This has proved to be quite a challenge to get working as testing was very laborious. I can confirm that it worked with the update from CrashPlan PRO 3.2 to 3.2.1 , and from CrashPlan 3.2.1 to 3.4.1:

CrashPlan-update-repair

 

Installation

  • This package is for Marvell Kirkwood, Marvell Armada 370/XP, Intel and Freescale QorIQ/PowerQUICC PowerPC CPUs only, so please check which CPU your NAS has. It will work on an unmodified NAS, no hacking or bootstrapping required. It will only work on older PowerQUICC PowerPC models that are running DSM 5.0. It is technically possible to run CrashPlan on older DSM versions, but it requires chroot-ing to a Debian install. Christophe from chreggy.fr has recently released packages to automate this.
  • In the User Control Panel in DSM, enable the User Homes service.
  • Install the package directly from Package Center in DSM. In Settings -> Package Sources add my package repository URL which is http://packages.pcloadletter.co.uk.
  • You will need to install either one of my Java SE Embedded packages first (Java 6 or 7). Read the instructions on that page carefully too.
  • If you previously installed CrashPlan manually using the Synology Wiki, you can find uninstall instructions here.
 

Notes

  • The package downloads the CrashPlan installer directly from Code 42 Software, following acceptance of their EULA. I am complying with their wish that no one redistributes it.
  • CrashPlan is installed in headless mode – backup engine only. This is configured by a desktop client, but operates independently of it.
  • The engine daemon script checks the amount of system RAM and scales the Java heap size appropriately (up to the default maximum of 512MB). This can be overridden in a persistent way if you are backing up very large backup sets by editing /volume1/@appstore/CrashPlan/syno_package.vars. If you’re considering buying a NAS purely to use CrashPlan and intend to back up more than a few hundred GB then I strongly advise buying one of the Intel models which come with 1GB RAM and can be upgraded to 3GB very cheaply. RAM is very limited on the ARM ones. 128MB RAM on the J series means CrashPlan is running with only one fifth of the recommended heap size, so I doubt it’s viable for backing up very much at all. My DS111 has 256MB of RAM and currently backs up around 60GB with no issues. I have found that a 512MB heap was insufficient to back up more than 2TB of files on a Windows server. It kept restarting the backup engine every few minutes until I increased the heap to 1024MB.
  • As with my other syno packages, the daemon user account password is randomized when it is created using the openssl binary. DSM Package Center runs as the root user so my script starts the package using an su command. This means that you can change the password yourself and CrashPlan will still work.
  • The default location for saving friends’ backups is set to /volume1/crashplan/backupArchives (where /volume1 is you primary storage volume) to eliminate the chance of them being destroyed accidentally by uninstalling the package.
  • The first time you run the server you will need to stop it and restart it before you can connect the client. This is because a config file that’s only created on first run needs to be edited by one of my scripts. The engine is then configured to listen on all interfaces on the default port 4243.
  • Once the engine is running, you can manage it by installing CrashPlan on another computer, and editing the file conf/ui.properties on that computer so that this line:
    #serviceHost=127.0.0.1
    is uncommented (by removing the hash symbol) and set to the IP address of your NAS, e.g.:
    serviceHost=192.168.1.210
    On Windows you can also disable the CrashPlan service if you will only use the client.
  • If you need to manage CrashPlan from a remote location, I suggest you do so using SSH tunnelling as per this support document.
  • The package supports upgrading to future versions while preserving the machine identity, logs, login details, and cache. Upgrades can now take place without requiring a login from the client afterwards.
  • If you remove the package completely and re-install it later, you can re-attach to previous backups. When you log in to the Desktop Client with your existing account after a re-install, you can select “adopt computer” to merge the records, and preserve your existing backups. I haven’t tested whether this also re-attaches links to friends’ CrashPlan computers and backup sets, though the latter does seem possible in the Friends section of the GUI. It’s probably a good idea to test that this survives a package reinstall before you start relying on it. Sometimes, particularly with CrashPlan PRO I think, the adopt option is not offered. In this case you can log into CrashPlan Central and retrieve your computer’s GUID. On the CrashPlan client, double-click on the logo in the top right and you’ll enter a command line mode. You can use the GUID command to change the system’s GUID to the one you just retrieved from your account.
  • The log which is displayed in the package’s Log tab is actually the activity history. If you’re trying to troubleshoot an issue you will need to use an SSH session to inspect the two engine log files which are:
    /volume1/@appstore/CrashPlan/log/engine_output.log
    /volume1/@appstore/CrashPlan/log/engine_error.log
  • When CrashPlan downloads and attempts to run an automatic update, the script will most likely fail and stop the package. This is typically caused by syntax differences with the Synology versions of certain Linux shell commands (like rm, mv, or ps). You will need to wait several minutes in the event of this happening before you take action, because the update script tries to restart CrashPlan 10 times at 10 second intervals. After this, you simply start the package again in Package Center and my scripts will fix the update, then run it. One final package restart is required before you can connect with the CrashPlan Desktop client (remember to update that too).
  • After their backup is seeded some users may wish to schedule the CrashPlan engine using cron so that it only runs at certain times. This is particularly useful on ARM systems because CrashPlan currently prevents hibernation while it is running (unresolved issue, reported to Code 42). To schedule, edit /etc/crontab and add the following entries for starting and stopping CrashPlan:
    55 2 * * * root /var/packages/CrashPlan/scripts/start-stop-status start
    0  4 * * * root /var/packages/CrashPlan/scripts/start-stop-status stop

    This example would configure CrashPlan to run daily between 02:55 and 04:00am. CrashPlan by default will scan the whole backup selection for changes at 3:00am so this is ideal. The simplest way to edit crontab if you’re not really confident with Linux is to install Merty’s Config File Editor package, which requires the official Synology Perl package to be installed too (since DSM 4.2). After editing crontab you will need to restart the cron daemon for the changes to take effect:
    /usr/syno/etc.defaults/rc.d/S04crond.sh stop
    /usr/syno/etc.defaults/rc.d/S04crond.sh start

    It is vitally important that you do not improvise your own startup commands or use a different account because this will most likely break the permissions on the config files, causing additional problems. The package scripts are designed to be run as root, and they will in turn invoke the CrashPlan engine using its own dedicated user account.
  • If you update DSM later, you will need to re-install the Java package or else UTF-8 and locale support will be broken by the update.
  • If you decide to sign up for one of CrashPlan’s paid backup services as a result of my work on this, I would really appreciate it if you could use this affiliate link, or consider donating using the PayPal button on the right.
 

Package scripts

For information, here are the package scripts so you can see what it’s going to do. You can get more information about how packages work by reading the Synology Package wiki.

installer.sh

#!/bin/sh

#--------CRASHPLAN installer script
#--------package maintained at pcloadletter.co.uk

DOWNLOAD_PATH="http://download.crashplan.com/installs/linux/install/${SYNOPKG_PKGNAME}"
[ "${SYNOPKG_PKGNAME}" == "CrashPlan" ] && DOWNLOAD_FILE="CrashPlan_3.6.3_Linux.tgz"
[ "${SYNOPKG_PKGNAME}" == "CrashPlanPRO" ] && DOWNLOAD_FILE="CrashPlanPRO_3.6.3_Linux.tgz"
[ "${SYNOPKG_PKGNAME}" == "CrashPlanPROe" ] && DOWNLOAD_FILE="CrashPlanPROe_3.6.3_Linux.tgz"
DOWNLOAD_URL="${DOWNLOAD_PATH}/${DOWNLOAD_FILE}"
CPI_FILE="${SYNOPKG_PKGNAME}_*.cpi"
EXTRACTED_FOLDER="${SYNOPKG_PKGNAME}-install"
DAEMON_USER="`echo ${SYNOPKG_PKGNAME} | awk {'print tolower($_)'}`"
DAEMON_PASS="`openssl rand 12 -base64 2>/dev/null`"
DAEMON_ID="${SYNOPKG_PKGNAME} daemon user"
DAEMON_HOME="/var/services/homes/${DAEMON_USER}"
OPTDIR="${SYNOPKG_PKGDEST}"
VARS_FILE="${OPTDIR}/install.vars"
ENGINE_SCRIPT="CrashPlanEngine"
SYNO_CPU_ARCH="`uname -m`"
[ "${SYNO_CPU_ARCH}" == "x86_64" ] && SYNO_CPU_ARCH="i686"
NATIVE_BINS_URL="http://packages.pcloadletter.co.uk/downloads/crashplan-native-${SYNO_CPU_ARCH}.tgz"   
NATIVE_BINS_FILE="`echo ${NATIVE_BINS_URL} | sed -r "s%^.*/(.*)%\1%"`"
INSTALL_FILES="${DOWNLOAD_URL} ${NATIVE_BINS_URL}"
TEMP_FOLDER="`find / -maxdepth 2 -name '@tmp' | head -n 1`"
#the Manifest folder is where friends' backup data is stored
#we set it outside the app folder so it persists after a package uninstall
MANIFEST_FOLDER="/`echo $TEMP_FOLDER | cut -f2 -d'/'`/crashplan"
LOG_FILE="${SYNOPKG_PKGDEST}/log/history.log.0"
UPGRADE_FILES="syno_package.vars conf/my.service.xml conf/service.login conf/service.model"
UPGRADE_FOLDERS="log cache"

source /etc/profile
PUBLIC_FOLDER="`cat /usr/syno/etc/smb.conf | sed -r '/\/public$/!d;s/^.*path=(\/volume[0-9]{1,4}\/public).*$/\1/'`"


preinst ()
{
  if [ -z ${PUBLIC_FOLDER} ]; then
    echo "A shared folder called 'public' could not be found - note this name is case-sensitive. "
    echo "Please create this using the Shared Folder DSM Control Panel and try again."
    exit 1
  fi

  if [ -z ${JAVA_HOME} ]; then
    echo "Java is not installed or not properly configured. JAVA_HOME is not defined. "
    echo "Download and install the Java Synology package from http://wp.me/pVshC-z5"
    exit 1
  fi
  
  if [ ! -f ${JAVA_HOME}/bin/java ]; then
    echo "Java is not installed or not properly configured. The Java binary could not be located. "
    echo "Download and install the Java Synology package from http://wp.me/pVshC-z5"
    exit 1
  fi
  
  #is the User Home service enabled?
  UH_SERVICE=maybe
  synouser --add userhometest Testing123 "User Home test user" 0 "" ""
  UHT_HOMEDIR=`cat /etc/passwd | sed -r '/User Home test user/!d;s/^.*:User Home test user:(.*):.*$/\1/'`
  if echo $UHT_HOMEDIR | grep '/var/services/homes/' > /dev/null; then
    if [ ! -d $UHT_HOMEDIR ]; then
      UH_SERVICE=false
    fi
  fi
  synouser --del userhometest
  #remove home directory (needed since DSM 4.1)
  [ -e /var/services/homes/userhometest ] && rm -r /var/services/homes/userhometest
  if [ "${UH_SERVICE}" == "false" ]; then
    echo "The User Home service is not enabled. Please enable this feature in the User control panel in DSM."
    exit 1
  fi
  
  cd ${TEMP_FOLDER}
  for WGET_URL in ${INSTALL_FILES}
  do
    WGET_FILENAME="`echo ${WGET_URL} | sed -r "s%^.*/(.*)%\1%"`"
    [ -f ${TEMP_FOLDER}/${WGET_FILENAME} ] && rm ${TEMP_FOLDER}/${WGET_FILENAME}
    wget ${WGET_URL}
    if [[ $? != 0 ]]; then
      if [ -d ${PUBLIC_FOLDER} ] && [ -f ${PUBLIC_FOLDER}/${WGET_FILENAME} ]; then
        cp ${PUBLIC_FOLDER}/${WGET_FILENAME} ${TEMP_FOLDER}
      else     
        echo "There was a problem downloading ${WGET_FILENAME} from the official download link, "
        echo "which was \"${WGET_URL}\" "
        echo "Alternatively, you may download this file manually and place it in the 'public' shared folder. "
        exit 1
      fi
    fi
  done
 
  exit 0
}


postinst ()
{
  #create daemon user
  synouser --add ${DAEMON_USER} ${DAEMON_PASS} "${DAEMON_ID}" 0 "" ""
  
  #save the daemon user's homedir as variable in that user's profile
  #this is needed because new users seem to inherit a HOME value of /root which they have no permissions for.
  su - ${DAEMON_USER} -s /bin/sh -c "echo export HOME=\'${DAEMON_HOME}\' >> .profile"

  #extract CPU-specific additional binaries
  mkdir ${SYNOPKG_PKGDEST}/bin
  cd ${SYNOPKG_PKGDEST}/bin
  tar xzf ${TEMP_FOLDER}/${NATIVE_BINS_FILE} && rm ${TEMP_FOLDER}/${NATIVE_BINS_FILE}

  #extract main archive
  cd ${TEMP_FOLDER}
  tar xzf ${TEMP_FOLDER}/${DOWNLOAD_FILE} && rm ${TEMP_FOLDER}/${DOWNLOAD_FILE} 
  
  #extract cpio archive
  cd ${SYNOPKG_PKGDEST}
  cat "${TEMP_FOLDER}/${EXTRACTED_FOLDER}"/${CPI_FILE} | gzip -d -c | ${SYNOPKG_PKGDEST}/bin/cpio -i --no-preserve-owner
  
  echo "#uncomment to expand Java max heap size beyond prescribed value (will survive upgrades)" > ${SYNOPKG_PKGDEST}/syno_package.vars
  echo "#you probably only want more than the recommended 512M if you're backing up extremely large volumes of files" >> ${SYNOPKG_PKGDEST}/syno_package.vars
  echo "#USR_MAX_HEAP=512M" >> ${SYNOPKG_PKGDEST}/syno_package.vars
  echo >> ${SYNOPKG_PKGDEST}/syno_package.vars

  #the following Package Center variables will need retrieving if launching CrashPlan via cron
  echo "CRON_SYNOPKG_PKGNAME='${SYNOPKG_PKGNAME}'" >> ${SYNOPKG_PKGDEST}/syno_package.vars
  echo "CRON_SYNOPKG_PKGDEST='${SYNOPKG_PKGDEST}'" >> ${SYNOPKG_PKGDEST}/syno_package.vars

  cp ${TEMP_FOLDER}/${EXTRACTED_FOLDER}/scripts/${ENGINE_SCRIPT} ${OPTDIR}/bin
  cp ${TEMP_FOLDER}/${EXTRACTED_FOLDER}/scripts/run.conf ${OPTDIR}/bin
  mkdir -p ${MANIFEST_FOLDER}/backupArchives    
  chown -R ${DAEMON_USER} ${MANIFEST_FOLDER}
  
  #save install variables which Crashplan expects its own installer script to create
  echo TARGETDIR=${SYNOPKG_PKGDEST} > ${VARS_FILE}
  echo BINSDIR=/bin >> ${VARS_FILE}
  echo MANIFESTDIR=${MANIFEST_FOLDER}/backupArchives >> ${VARS_FILE}
  #leave these ones out which should help upgrades from Code42 to work (based on examining an upgrade script)
  #echo INITDIR=/etc/init.d >> ${VARS_FILE}
  #echo RUNLVLDIR=/usr/syno/etc/rc.d >> ${VARS_FILE}
  echo INSTALLDATE=`date +%Y%m%d` >> ${VARS_FILE}
  echo JAVACOMMON=\${JAVA_HOME}/bin/java >> ${VARS_FILE}
  cat ${TEMP_FOLDER}/${EXTRACTED_FOLDER}/install.defaults >> ${VARS_FILE}
  
  #remove temp files
  rm -r ${TEMP_FOLDER}/${EXTRACTED_FOLDER}
  
  #change owner of CrashPlan folder tree
  chown -R ${DAEMON_USER} ${SYNOPKG_PKGDEST}
  
  exit 0
}


preuninst ()
{
  #make sure engine is stopped
  su - ${DAEMON_USER} -s /bin/sh -c "${OPTDIR}/bin/${ENGINE_SCRIPT} stop"
  sleep 2
  
  exit 0
}


postuninst ()
{
  if [ -f ${SYNOPKG_PKGDEST}/syno_package.vars ]; then
    source ${SYNOPKG_PKGDEST}/syno_package.vars
  fi

  if [ "${LIBFFI_SYMLINK}" == "YES" ]; then
    rm /lib/libffi.so.5
  fi
  
  #if it doesn't exist, but is still a link then it's a broken link and should also be deleted
  if [ ! -e /lib/libffi.so.5 ]; then
    [ -L /lib/libffi.so.5 ] && rm /lib/libffi.so.5
  fi
    
  #remove daemon user
  synouser --del ${DAEMON_USER}
  
  #remove daemon user's home directory (needed since DSM 4.1)
  [ -e /var/services/homes/${DAEMON_USER} ] && rm -r /var/services/homes/${DAEMON_USER}
  
 exit 0
}

preupgrade ()
{
  #make sure engine is stopped
  su - ${DAEMON_USER} -s /bin/sh -c "${OPTDIR}/bin/${ENGINE_SCRIPT} stop"
  sleep 2
  
  #if identity and config data exists back it up
  if [ -d ${DAEMON_HOME}/.crashplan ]; then
    mkdir -p ${SYNOPKG_PKGDEST}/../${DAEMON_USER}_data_mig/conf
    mv ${DAEMON_HOME}/.crashplan ${SYNOPKG_PKGDEST}/../${DAEMON_USER}_data_mig
    for FILE_TO_MIGRATE in ${UPGRADE_FILES}; do
      if [ -f ${OPTDIR}/${FILE_TO_MIGRATE} ]; then
        cp ${OPTDIR}/${FILE_TO_MIGRATE} ${SYNOPKG_PKGDEST}/../${DAEMON_USER}_data_mig/${FILE_TO_MIGRATE}
      fi
    done
    for FOLDER_TO_MIGRATE in ${UPGRADE_FOLDERS}; do
      if [ -d ${OPTDIR}/${FOLDER_TO_MIGRATE} ]; then
        mv ${OPTDIR}/${FOLDER_TO_MIGRATE} ${SYNOPKG_PKGDEST}/../${DAEMON_USER}_data_mig
      fi
    done
  fi

  exit 0
}


postupgrade ()
{
  #use the migrated identity and config data from the previous version
  if [ -d ${SYNOPKG_PKGDEST}/../${DAEMON_USER}_data_mig/.crashplan ]; then
    mv ${SYNOPKG_PKGDEST}/../${DAEMON_USER}_data_mig/.crashplan ${DAEMON_HOME}
    for FILE_TO_MIGRATE in ${UPGRADE_FILES}; do
      if [ -f ${SYNOPKG_PKGDEST}/../${DAEMON_USER}_data_mig/${FILE_TO_MIGRATE} ]; then
        mv ${SYNOPKG_PKGDEST}/../${DAEMON_USER}_data_mig/${FILE_TO_MIGRATE} ${OPTDIR}/${FILE_TO_MIGRATE}
      fi
    done
    for FOLDER_TO_MIGRATE in ${UPGRADE_FOLDERS}; do
    if [ -d ${SYNOPKG_PKGDEST}/../${DAEMON_USER}_data_mig/${FOLDER_TO_MIGRATE} ]; then
      mv ${SYNOPKG_PKGDEST}/../${DAEMON_USER}_data_mig/${FOLDER_TO_MIGRATE} ${OPTDIR}
    fi
    done
    rmdir ${SYNOPKG_PKGDEST}/../${DAEMON_USER}_data_mig/conf
    rmdir ${SYNOPKG_PKGDEST}/../${DAEMON_USER}_data_mig
    
    #make CrashPlan log entry
    TIMESTAMP="`date +%D` `date +%I:%M%p`"
    echo "I ${TIMESTAMP} Synology Package Center updated ${SYNOPKG_PKGNAME} to version ${SYNOPKG_PKGVER}" >> ${LOG_FILE}
    
    #daemon user has been deleted and recreated so we need to reset ownership (new UID)
    chown -R ${DAEMON_USER} ${DAEMON_HOME}/.crashplan
    chown -R ${DAEMON_USER} ${SYNOPKG_PKGDEST}
    
    #read manifest location from the migrated XML config, and reset ownership on that path too
    if [ -f ${SYNOPKG_PKGDEST}/conf/my.service.xml ]; then
      MANIFEST_FOLDER=`cat ${SYNOPKG_PKGDEST}/conf/my.service.xml | grep "<manifestPath>" | cut -f2 -d'>' | cut -f1 -d'<'`
      chown -R ${DAEMON_USER} ${MANIFEST_FOLDER}
    fi
    
    #the following Package Center variables will need retrieving if launching CrashPlan via cron
    grep "^CRON_SYNOPKG_PKGNAME" ${SYNOPKG_PKGDEST}/syno_package.vars > /dev/null \
     || echo "CRON_SYNOPKG_PKGNAME='${SYNOPKG_PKGNAME}'" >> ${SYNOPKG_PKGDEST}/syno_package.vars
    grep "^CRON_SYNOPKG_PKGDEST" ${SYNOPKG_PKGDEST}/syno_package.vars > /dev/null \
     || echo "CRON_SYNOPKG_PKGDEST='${SYNOPKG_PKGDEST}'" >> ${SYNOPKG_PKGDEST}/syno_package.vars
  fi
  
  exit 0
}
 

start-stop-status.sh

#!/bin/sh

#--------CRASHPLAN start-stop-status script
#--------package maintained at pcloadletter.co.uk

if [ "${SYNOPKG_PKGNAME}" == "" ]; then
  #if this script has been invoked by cron then some Package Center vars are undefined
  source "`dirname $0`/../target/syno_package.vars"
  SYNOPKG_PKGNAME="${CRON_SYNOPKG_PKGNAME}" 
  SYNOPKG_PKGDEST="${CRON_SYNOPKG_PKGDEST}"
  CRON_LAUNCHED=True
fi

#Main variables section
DAEMON_USER="`echo ${SYNOPKG_PKGNAME} | awk {'print tolower($_)'}`"
DAEMON_HOME="/var/services/homes/${DAEMON_USER}"
OPTDIR="${SYNOPKG_PKGDEST}"
TEMP_FOLDER="`find / -maxdepth 2 -name '@tmp' | head -n 1`"
MANIFEST_FOLDER="/`echo $TEMP_FOLDER | cut -f2 -d'/'`/crashplan"
LOG_FILE="${SYNOPKG_PKGDEST}/log/history.log.0"
ENGINE_SCRIPT="CrashPlanEngine"
APP_NAME="CrashPlanService"
SCRIPTS_TO_EDIT="${ENGINE_SCRIPT}"
ENGINE_CFG="run.conf"
LIBFFI_SO_NAMES="5 6" #armada370 build of libjnidispatch.so is newer, and uses libffi.so.6
CFG_PARAM="SRV_JAVA_OPTS"
source ${OPTDIR}/install.vars

JAVA_MIN_HEAP=`grep "^${CFG_PARAM}=" "${OPTDIR}/bin/${ENGINE_CFG}" | sed -r "s/^.*-Xms([0-9]+)[Mm] .*$/\1/"`
SYNO_CPU_ARCH="`uname -m`"


case $1 in
  start)    
    #set the current timezone for Java so that log timestamps are accurate
    #we need to use the modern timezone names so that Java can figure out DST 
    SYNO_TZ=`cat /etc/synoinfo.conf | grep timezone | cut -f2 -d'"'`
    SYNO_TZ=`grep "^${SYNO_TZ}" /usr/share/zoneinfo/Timezone/tzname | sed -e "s/^.*= //"`
    grep "^export TZ" ${DAEMON_HOME}/.profile > /dev/null \
     && sed -i "s%^export TZ=.*$%export TZ='${SYNO_TZ}'%" ${DAEMON_HOME}/.profile \
     || echo export TZ=\'${SYNO_TZ}\' >> ${DAEMON_HOME}/.profile
    #this package stores the machine identity in the daemon user home directory
    #so we need to remove any old config data from previous manual installations or startups
    [ -d /var/lib/crashplan ] && rm -r /var/lib/crashplan

    #check persistent variables from syno_package.vars
    USR_MAX_HEAP=0
    if [ -f ${SYNOPKG_PKGDEST}/syno_package.vars ]; then
      source ${SYNOPKG_PKGDEST}/syno_package.vars
    fi
    USR_MAX_HEAP=`echo $USR_MAX_HEAP | sed -e "s/[mM]//"`

    #create or repair libffi symlink if a DSM upgrade has removed it
    for FFI_VER in ${LIBFFI_SO_NAMES}; do 
      if [ -e ${OPTDIR}/lib/libffi.so.${FFI_VER} ]; then
        if [ ! -e /lib/libffi.so.${FFI_VER} ]; then
          #if it doesn't exist, but is still a link then it's a broken link and should be deleted
          [ -L /lib/libffi.so.${FFI_VER} ] && rm /lib/libffi.so.${FFI_VER}
          ln -s ${OPTDIR}/lib/libffi.so.${FFI_VER} /lib/libffi.so.${FFI_VER}
        fi
      fi
    done

    #fix up some of the binary paths and fix some command syntax for busybox 
    #moved this to start-stop-status from installer.sh because Code42 push updates and these
    #new scripts will need this treatment too
    FIND_TARGETS=
    for TARGET in ${SCRIPTS_TO_EDIT}; do
      FIND_TARGETS="${FIND_TARGETS} -o -name ${TARGET}"
    done
    find ${OPTDIR} \( -name \*.sh ${FIND_TARGETS} \) | while IFS="" read -r FILE_TO_EDIT; do
      if [ -e ${FILE_TO_EDIT} ]; then
        #this list of substitutions will probably need expanding as new CrashPlan updates are released
        sed -i "s%^#!/bin/bash%#!${SYNOPKG_PKGDEST}/bin/bash%" "${FILE_TO_EDIT}"
        sed -i -r "s%(^\s*)nice -n%\1${SYNOPKG_PKGDEST}/bin/nice -n%" "${FILE_TO_EDIT}"
        sed -i -r "s%(^\s*)(/bin/ps|ps) [^\|]*\|%\1/bin/ps w \|%" "${FILE_TO_EDIT}"
        sed -i -r "s%\`ps [^\|]*\|%\`ps w \|%" "${FILE_TO_EDIT}"
        sed -i "s/rm -fv/rm -f/" "${FILE_TO_EDIT}"
        sed -i "s/mv -fv/mv -f/" "${FILE_TO_EDIT}"
      fi
    done

    #any downloaded upgrade script will usually have failed until the above changes are made so we need to
    #find it and start it, if it exists
    UPGRADE_SCRIPT=`find ${OPTDIR}/upgrade -name "upgrade.sh"`
    if [ -n "${UPGRADE_SCRIPT}" ]; then
      rm ${OPTDIR}/${ENGINE_SCRIPT}.pid
      SCRIPT_HOME=`dirname $UPGRADE_SCRIPT`

      #make CrashPlan log entry
      TIMESTAMP="`date +%D` `date +%I:%M%p`"
      echo "I ${TIMESTAMP} Synology repairing upgrade in ${SCRIPT_HOME}" >> ${LOG_FILE}

      mv ${SCRIPT_HOME}/upgrade.log ${SCRIPT_HOME}/upgrade.log.old
      chown -R ${DAEMON_USER} ${SYNOPKG_PKGDEST}
      su - ${DAEMON_USER} -s /bin/sh -c "cd ${SCRIPT_HOME} ; . upgrade.sh"
      mv ${SCRIPT_HOME}/upgrade.sh ${SCRIPT_HOME}/upgrade.sh.old
      exit 0
    fi

    #updates may also overwrite our native binaries
    if [ "${SYNO_CPU_ARCH}" == "x86_64" ]; then
      cp ${SYNOPKG_PKGDEST}/bin/synology-x86-glibc-2.4-shim.so ${OPTDIR}/lib
    else    
      cp -f ${SYNOPKG_PKGDEST}/bin/libjtux.so ${OPTDIR}
      cp -f ${SYNOPKG_PKGDEST}/bin/jna-3.2.5.jar ${OPTDIR}/lib
      cp -f ${SYNOPKG_PKGDEST}/bin/libffi.so.* ${OPTDIR}/lib
    fi

    #set appropriate Java max heap size
    RAM=$((`free | grep Mem: | sed -e "s/^ *Mem: *\([0-9]*\).*$/\1/"`/1024))
    if [ $RAM -le 128 ]; then
      JAVA_MAX_HEAP=80
    elif [ $RAM -le 256 ]; then
      JAVA_MAX_HEAP=192
    elif [ $RAM -le 512 ]; then
      JAVA_MAX_HEAP=384
    #CrashPlan's default max heap is 512MB
    elif [ $RAM -gt 512 ]; then
      JAVA_MAX_HEAP=512
    fi
    if [ $USR_MAX_HEAP -gt $JAVA_MAX_HEAP ]; then
      JAVA_MAX_HEAP=${USR_MAX_HEAP}
    fi   
    if [ $JAVA_MAX_HEAP -lt $JAVA_MIN_HEAP ]; then
      #can't have a max heap lower than min heap (ARM low RAM systems)
      $JAVA_MAX_HEAP=$JAVA_MIN_HEAP
    fi
    sed -i -r "s/(^${CFG_PARAM}=.*) -Xmx[0-9]+[mM] (.*$)/\1 -Xmx${JAVA_MAX_HEAP}m \2/" "${OPTDIR}/bin/${ENGINE_CFG}"
    
    #disable the use of the x86-optimized external Fast MD5 library if running on ARM and QorIQ CPUs
    #seems to be the default behaviour now but that may change again
    if [ "${SYNO_CPU_ARCH}" != "x86_64" ]; then
      grep "^${CFG_PARAM}=.*c42\.native\.md5\.enabled" "${OPTDIR}/bin/${ENGINE_CFG}" > /dev/null \
       || sed -i -r "s/(^${CFG_PARAM}=\".*)\"$/\1 -Dc42.native.md5.enabled=false\"/" "${OPTDIR}/bin/${ENGINE_CFG}"
    fi

    #move the Java temp directory from the default of /tmp
    grep "^${CFG_PARAM}=.*Djava\.io\.tmpdir" "${OPTDIR}/bin/${ENGINE_CFG}" > /dev/null \
     || sed -i -r "s%(^${CFG_PARAM}=\".*)\"$%\1 -Djava.io.tmpdir=${TEMP_FOLDER}\"%" "${OPTDIR}/bin/${ENGINE_CFG}"

    #reset ownership of all files to daemon user, so that manual edits to config files won't cause problems
    chown -R ${DAEMON_USER} ${SYNOPKG_PKGDEST}
    chown -R ${DAEMON_USER} ${DAEMON_HOME}    

    #now edit the XML config file, which only exists after first run
    if [ -f ${SYNOPKG_PKGDEST}/conf/my.service.xml ]; then

      #allow direct connections from CrashPlan Desktop client on remote systems
      #you must edit the value of serviceHost in conf/ui.properties on the client you connect with
      #users report that this value is sometimes reset so now it's set every service startup 
      sed -i "s/<serviceHost>127\.0\.0\.1<\/serviceHost>/<serviceHost>0\.0\.0\.0<\/serviceHost>/" "${SYNOPKG_PKGDEST}/conf/my.service.xml"
      
      #this change is made only once in case you want to customize the friends' backup location
      if [ "${MANIFEST_PATH_SET}" != "True" ]; then

        #keep friends' backup data outside the application folder to make accidental deletion less likely 
        sed -i "s%<manifestPath>.*</manifestPath>%<manifestPath>${MANIFEST_FOLDER}/backupArchives/</manifestPath>%" "${SYNOPKG_PKGDEST}/conf/my.service.xml"
        echo "MANIFEST_PATH_SET=True" >> ${SYNOPKG_PKGDEST}/syno_package.vars
      fi

      #since CrashPlan version 3.5.3 the value javaMemoryHeapMax also needs setting to match that used in bin/run.conf
      sed -i -r "s%(<javaMemoryHeapMax>)[0-9]+[mM](</javaMemoryHeapMax>)%\1${JAVA_MAX_HEAP}m\2%" "${SYNOPKG_PKGDEST}/conf/my.service.xml"
    else
      echo "Wait a few seconds, then stop and restart the package to allow desktop client connections." > "${SYNOPKG_TEMP_LOGFILE}"
    fi
    if [ "${CRON_LAUNCHED}" == "True" ]; then
      [ -e /var/packages/${SYNOPKG_PKGNAME}/enabled ] || touch /var/packages/${SYNOPKG_PKGNAME}/enabled
    fi

    #delete any stray Java temp files
    find /tmp -name "jna*.tmp" -user ${DAEMON_USER} | while IFS="" read -r FILE_TO_DEL; do
      if [ -e ${FILE_TO_DEL} ]; then
        rm ${FILE_TO_DEL}
      fi
    done

    #increase the system-wide maximum number of open files from Synology default of 24466
    echo "65536" > /proc/sys/fs/file-max

    #raise the maximum open file count from the Synology default of 1024 - thanks Casper K. for figuring this out
    #http://support.code42.com/Administrator/3.6_And_4.0/Troubleshooting/Too_Many_Open_Files
    ulimit -n 65536

    if [ "${SYNO_CPU_ARCH}" == "x86_64" ]; then
      #Intel synos running older DSM need rwojo's glibc version shim for inotify support
      #https://github.com/wojo/synology-x86-glibc-2.4-shim
      GLIBC_VER="`/lib/libc.so.6 | grep -m 1 version | sed -r "s/^[^0-9]*([0-9].*[0-9])\,.*$/\1/"`"
      if [ "${GLIBC_VER}" == "2.3.6" ]; then
        su - ${DAEMON_USER} -s /bin/sh -c "LD_PRELOAD=${SYNOPKG_PKGDEST}/lib/synology-x86-glibc-2.4-shim.so ${OPTDIR}/bin/${ENGINE_SCRIPT} start"
      else
        su - ${DAEMON_USER} -s /bin/sh -c "${OPTDIR}/bin/${ENGINE_SCRIPT} start"
      fi
    else
      su - ${DAEMON_USER} -s /bin/sh -c "${OPTDIR}/bin/${ENGINE_SCRIPT} start"
    fi
    exit 0
  ;;

  stop)
    su - ${DAEMON_USER} -s /bin/sh -c "${OPTDIR}/bin/${ENGINE_SCRIPT} stop"
    if [ "${CRON_LAUNCHED}" == "True" ]; then
      [ -e /var/packages/${SYNOPKG_PKGNAME}/enabled ] && rm /var/packages/${SYNOPKG_PKGNAME}/enabled
    fi
    exit 0
  ;;

  status)
    PID=`/bin/ps w| grep "app=${APP_NAME}" | grep -v grep | awk '{ print $1 }'`
    if [ -n "$PID" ]; then
      exit 0
    else
      exit 1
    fi
  ;;

  log)
    echo "${LOG_FILE}"
    exit 0
  ;;
esac
 

Changelog:

  • 0027 Fixed open file handle limit for very large backup sets (ulimit fix)
  • 0026 Updated all CrashPlan clients to version 3.6.3, improved handling of Java temp files
  • 0025 glibc version shim no longer used on Intel Synology models running DSM 5.0
  • 0024 Updated to CrashPlan PROe 3.6.1.4 and added support for PowerPC 2010 Synology models running DSM 5.0
  • 0023 Added support for Intel Atom Evansport and Armada XP CPUs in new DSx14 products
  • 0022 Updated all CrashPlan client versions to 3.5.3, compiled native binary dependencies to add support for Armada 370 CPU (DS213j), start-stop-status.sh now updates the new javaMemoryHeapMax value in my.service.xml to the value defined in syno_package.vars
  • 0021 Updated CrashPlan to version 3.5.2
  • 0020 Fixes for DSM 4.2
  • 018 Updated CrashPlan PRO to version 3.4.1
  • 017 Updated CrashPlan and CrashPlan PROe to version 3.4.1, and improved in-app update handling
  • 016 Added support for Freescale QorIQ CPUs in some x13 series Synology models, and installer script now downloads native binaries separately to reduce repo hosting bandwidth, PowerQUICC PowerPC processors in previous Synology generations with older glibc versions are not supported
  • 015 Added support for easy scheduling via cron – see updated Notes section
  • 014 DSM 4.1 user profile permissions fix
  • 013 implemented update handling for future automatic updates from Code 42, and incremented CrashPlanPRO client to release version 3.2.1
  • 012 incremented CrashPlanPROe client to release version 3.3
  • 011 minor fix to allow a wildcard on the cpio archive name inside the main installer package (to fix CP PROe client since Code 42 Software had amended the cpio file version to 3.2.1.2)
  • 010 minor bug fix relating to daemon home directory path
  • 009 rewrote the scripts to be even easier to maintain and unified as much as possible with my imminent CrashPlan PROe server package, fixed a timezone bug (tightened regex matching), moved the script-amending logic from installer.sh to start-stop-status.sh with it now applying to all .sh scripts each startup so perhaps updates from Code42 might work in future, if wget fails to fetch the installer from Code42 the installer will look for the file in the public shared folder
  • 008 merged the 14 package scripts each (7 for ARM, 7 for Intel) for CP, CP PRO, & CP PROe – 42 scripts in total – down to just two! ARM & Intel are now supported by the same package, Intel synos now have working inotify support (Real-Time Backup) thanks to rwojo’s shim to pass the glibc version check, upgrade process now retains login, cache and log data (no more re-scanning), users can specify a persistent larger max heap size for very large backup sets
  • 007 fixed a bug that broke CrashPlan if the Java folder moved (if you changed version)
  • 006 installation now fails without User Home service enabled, fixed Daylight Saving Time support, automated replacing the ARM libffi.so symlink which is destroyed by DSM upgrades, stopped assuming the primary storage volume is /volume1, reset ownership on /var/lib/crashplan and the Friends backup location after installs and upgrades
  • 005 added warning to restart daemon after 1st run, and improved upgrade process again
  • 004 updated to CrashPlan 3.2.1 and improved package upgrade process, forced binding to 0.0.0.0 each startup
  • 003 fixed ownership of /volume1/crashplan folder
  • 002 updated to CrashPlan 3.2
  • 001 intial public release
 
 

Minecraft package for Synology NAS

UPDATE – The instructions and notes on this page apply to both the Minecraft and the CraftBukkit packages hosted on my repo. Now works on QorIQ CPU Synology models!

Minecraft is an intriguing game. I only recently bought it after a friend had recommended it. There’s a lot to like: the striking retro visual style, the mellow sporadic music, exploring the procedurally generated world, the logical way items are crafted, multiplayer collaboration, and emergent gameplay. Notch its creator certainly deserves the fortune he’s now sitting on. I thoroughly recommend you watch this short documentary all about it.

Minecraft Package Info

Right after I released the Java package for Synology a couple of people mentioned that they wanted to use it for installing a Minecraft server. Now that I have the game, I decided to try that too. I had assumed that the ARM CPUs would not be capable of running it, and the first tests seemed to confirm this. I decided it would be worth creating a package nonetheless since the Intel CPU NAS units would be ok, especially with their extra RAM. However, with a little performance tuning it actually runs acceptably (no lag when mining blocks) on my ARM powered DS111 which only has 256MB. The CPU load is 100% for a while at first, but soon after playing it settles down to around 60%. I have briefly tested with two players connected and after a bit of lag at first (one second delay to mine a block) it does seem to settle down.

Minecraft running showing draw distance and server load

Minecraft showing draw distance and server load running on a Synology DS111

 

Installation

  • This package is not CPU specific. If you have Java it will run. At the moment that means ARM, Intel and QorIQ PowerPC processors, but not the older PowerPCs.
  • In the User Control Panel in DSM, enable the User Homes service.
  • Install the package directly from Package Center in DSM. In Settings -> Package Sources add my package repository URL which is http://packages.pcloadletter.co.uk. You will need to install either one of my Java SE for Embedded packages first (Java 6 or 7).

Notes

  • The package fetches the minecraft server jar file from Mojang as it is installed. I am complying with their wish that no one redistributes it.
  • The server daemon script checks the amount of system RAM and scales the Java heap size appropriately. It also applies a few performance tweaks to Java to try to reduce garbage collection latency.
  • The first time you run the server it will create a new world, which can take a few minutes. You cannot interrupt this but you can check on progress by repeatedly viewing the Log tab.
  • Because the time investments in playing Minecraft can be so considerable, when you uninstall the package it will back up the world folder and settings to /volume1/public/minecraftworld.todaysdate.bak to prevent accidental deletion.
  • The package supports upgrades to future versions while preserving the world folder and server settings.
  • If you want to transplant an existing world folder into the server, copy it to /volume1/@appstore/Minecraft. You will also need to run chown -R minecraft /volume1/@appstore/Minecraft to grant ownership of the files to the daemon user.
  • The first time you run the package, the server config file /volume1/@appstore/Minecraft/server.properties is generated. The next time it is launched, my script reduces the default draw distance for ARM CPUs from 10 chunks to 7. This was appropriate for my DS111 to prevent latency when mining blocks, but you may wish to reduce this further on the J series NAS units which have less RAM and slower CPUs. Other server files (white-list.txt etc.) are found in the same folder. For CraftBukkit the files are in the folder /volume1/@appstore/Craftbukkit/server.properties (note the capitalization – the CraftBukkit project seemed to change this after I had already created the package with a lower case ‘b’).
  • The simplest way to edit these config files if you’re not really confident with Linux is to install Merty’s Config File Editor package, which requires the official Synology Perl package to be installed too (since DSM 4.2). Load Config File Editor, then in the dropdown menu edit Config File Editor’s own config (it’s the last in the list) and add the lines:
    /volume1/@appstore/Minecraft/server.properties,Minecraft-properties
    /volume1/@appstore/Minecraft/white-list.txt,Minecraft-whitelist
    /volume1/@appstore/Minecraft/ops.txt,Minecraft-ops

    Make sure to add an extra blank line underneath, save, then relaunch CFE and you’ll have entries for Minecraft in the dropdown. You’ll need to restart the Minecraft package for any changes to take effect.
  • It was a bit tricky to get the server to shut down gracefully without just killing the Java process. We need it to shutdown properly so it saves the active chunks to disk first. Most of the guides on the Net use the screen binary which isn’t included with Synology DSM, and I didn’t want to have to make a version of the package for each CPU architecture. I found that I could use tail to send the last line of the file /tmp/stdin.minecraft to the server (/tmp/stdin.craftbukkit for the CraftBukkit package). This is how the stop command is issued. You could send your own commands, for instance echo say Hello players >> /tmp/stdin.minecraft. You can verify that the command was received by looking at the server log in Package Center.
  • The server runs on the default TCP port for Minecraft (25565) so you will need to port forward this on your router if you want it to be publicly accessible.
 

Package scripts

For information, here are the package scripts so you can see what it’s going to do. You can get more information about how packages work by reading the Synology Package wiki.

installer.sh

#!/bin/sh

#--------MINECRAFT/CRAFTBUKKIT installer script
#--------package maintained at pcloadletter.co.uk

if [ "${SYNOPKG_PKGNAME}" == "Minecraft" ]; then
  DOWNLOAD_PATH="http://s3.amazonaws.com/MinecraftDownload/launcher"
  DOWNLOAD_FILE="minecraft_server.jar"
  UPGRADE_FILES="server.properties *.txt world"
fi
if [ "${SYNOPKG_PKGNAME}" == "Craftbukkit" ]; then
  DOWNLOAD_PATH="http://cbukk.it"
  DOWNLOAD_FILE="craftbukkit-beta.jar"
  UPGRADE_FILES="server.properties *.txt *.yml world world_nether world_the_end plugins bukkit_update"
fi

DOWNLOAD_URL="${DOWNLOAD_PATH}/${DOWNLOAD_FILE}"
DAEMON_USER="`echo ${SYNOPKG_PKGNAME} | awk {'print tolower($_)'}`"
DAEMON_ID="${SYNOPKG_PKGNAME} daemon user"
DAEMON_PASS="`openssl rand 12 -base64 2>/dev/null`"
MIGRATION_FOLDER="${DAEMON_USER}_data_mig"
ENGINE_SCRIPT="/var/packages/${SYNOPKG_PKGNAME}/scripts/launcher.sh"
INSTALL_FILES="${DOWNLOAD_URL}"
source /etc/profile
TEMP_FOLDER="`find / -maxdepth 2 -name '@tmp' | head -n 1`"
PRIMARY_VOLUME="/`echo $TEMP_FOLDER | cut -f2 -d'/'`"
WORLD_BACKUP="${PRIMARY_VOLUME}/public/${DAEMON_USER}world.`date +\"%d-%b\"`.bak"

preinst ()
{
  if [ -z ${JAVA_HOME} ]; then
    echo "Java is not installed or not properly configured. JAVA_HOME is not defined. "
    echo "Download and install the Java Synology package from http://wp.me/pVshC-z5"
    exit 1
  fi
  
  if [ ! -f ${JAVA_HOME}/bin/java ]; then
    echo "Java is not installed or not properly configured. The Java binary could not be located. "
    echo "Download and install the Java Synology package from http://wp.me/pVshC-z5"
    exit 1
  fi
  
  #is the User Home service enabled?
  UH_SERVICE=maybe
  synouser --add userhometest Testing123 "User Home test user" 0 "" ""
  UHT_HOMEDIR=`cat /etc/passwd | sed -r '/User Home test user/!d;s/^.*:User Home test user:(.*):.*$/\1/'`
  if echo $UHT_HOMEDIR | grep '/var/services/homes/' > /dev/null; then
    if [ ! -d $UHT_HOMEDIR ]; then
      UH_SERVICE=false
    fi
  fi
  synouser --del userhometest
  #remove home directory (needed since DSM 4.1)
  [ -e /var/services/homes/userhometest ] && rm -r /var/services/homes/userhometest
  if [ ${UH_SERVICE} == "false" ]; then
    echo "The User Home service is not enabled. Please enable this feature in the User control panel in DSM."
    exit 1
  fi

  cd ${TEMP_FOLDER}
  for WGET_URL in ${INSTALL_FILES}
  do
    WGET_FILENAME="`echo ${WGET_URL} | sed -r "s%^.*/(.*)%\1%"`"
    [ -f ${TEMP_FOLDER}/${WGET_FILENAME} ] && rm ${TEMP_FOLDER}/${WGET_FILENAME}
    wget ${WGET_URL}
    if [[ $? != 0 ]]; then
      if [ -d ${PUBLIC_FOLDER} ] && [ -f ${PUBLIC_FOLDER}/${WGET_FILENAME} ]; then
        cp ${PUBLIC_FOLDER}/${WGET_FILENAME} ${TEMP_FOLDER}
      else     
        echo "There was a problem downloading ${WGET_FILENAME} from the official download link, "
        echo "which was \"${WGET_URL}\" "
        echo "Alternatively, you may download this file manually and place it in the 'public' shared folder. "
        exit 1
      fi
    fi
  done
  
  exit 0
}


postinst ()
{
  #create daemon user
  synouser --add ${DAEMON_USER} ${DAEMON_PASS} "${DAEMON_ID}" 0 "" ""
  
  mv ${TEMP_FOLDER}/${DAEMON_USER}*.jar ${SYNOPKG_PKGDEST}/${DAEMON_USER}.jar
  
  #determine the daemon user homedir and save that variable in the user's profile
  #this is needed because new users seem to inherit a HOME value of /root which they have no permissions for
  DAEMON_HOME="`cat /etc/passwd | grep "${DAEMON_ID}" | cut -f6 -d':'`"
  su - ${DAEMON_USER} -s /bin/sh -c "echo export HOME=\'${DAEMON_HOME}\' >> .profile"
  
  #change owner of folder tree
  chown -R ${DAEMON_USER} ${SYNOPKG_PKGDEST}
  
  exit 0
}


preuninst ()
{
  #make sure server is stopped
  su - ${DAEMON_USER} -s /bin/sh -c "${ENGINE_SCRIPT} stop ${SYNOPKG_PKGNAME} ${SYNOPKG_PKGDEST}"
  sleep 10
  
  #if a world exists, back it up to the public folder, just in case...
  if [ -d ${SYNOPKG_PKGDEST}/world ]; then
    if [ ! -d ${WORLD_BACKUP} ]; then
      mkdir -p ${WORLD_BACKUP}
    fi
    for ITEM in ${UPGRADE_FILES}; do
      mv ${SYNOPKG_PKGDEST}/${ITEM} ${WORLD_BACKUP}
    done
  fi
  
  exit 0
}


postuninst ()
{
  #remove daemon user
  synouser --del ${DAEMON_USER}
  
  #remove daemon user's home directory (needed since DSM 4.1)
  [ -e /var/services/homes/${DAEMON_USER} ] && rm -r /var/services/homes/${DAEMON_USER}
  
  exit 0
}


preupgrade ()
{
  #make sure the server is stopped
  su - ${DAEMON_USER} -s /bin/sh -c "${ENGINE_SCRIPT} stop ${SYNOPKG_PKGNAME} ${SYNOPKG_PKGDEST}"
  sleep 10
  
  #if a world exists, back it up
  if [ -d ${SYNOPKG_PKGDEST}/world ]; then
    mkdir ${SYNOPKG_PKGDEST}/../${MIGRATION_FOLDER}
    for ITEM in ${UPGRADE_FILES}; do
      if [ -e ${SYNOPKG_PKGDEST}/${ITEM} ]; then
        mv ${SYNOPKG_PKGDEST}/${ITEM} ${SYNOPKG_PKGDEST}/../${MIGRATION_FOLDER}
      fi
    done
  fi
  
  exit 0
}


postupgrade ()
{
  #use the migrated data files from the previous version
  if [ -d ${SYNOPKG_PKGDEST}/../${MIGRATION_FOLDER}/world ]; then
    for ITEM in ${UPGRADE_FILES}; do
      if [ -e ${SYNOPKG_PKGDEST}/../${MIGRATION_FOLDER}/${ITEM} ]; then
        mv ${SYNOPKG_PKGDEST}/../${MIGRATION_FOLDER}/${ITEM} ${SYNOPKG_PKGDEST}
      fi
    done
    rmdir ${SYNOPKG_PKGDEST}/../${MIGRATION_FOLDER}
    
    #daemon user has been deleted and recreated so we need to reset ownership (new UID)
    chown -R ${DAEMON_USER} ${SYNOPKG_PKGDEST}
  fi
  	
  exit 0
}
 

start-stop-status.sh

#!/bin/sh

#--------MINECRAFT/CRAFTBUKKIT start-stop-status script
#--------package maintained at pcloadletter.co.uk

DAEMON_USER="`echo ${SYNOPKG_PKGNAME} | awk {'print tolower($_)'}`"
DAEMON_ID="${SYNOPKG_PKGNAME} daemon user"
ENGINE_SCRIPT="/var/packages/${SYNOPKG_PKGNAME}/scripts/launcher.sh"
DAEMON_USER_SHORT=`echo ${DAEMON_USER} | cut -c 1-8`

daemon_status ()
{
    ps | grep "^ *[0-9]* ${DAEMON_USER_SHORT} .*java" > /dev/null
}

case $1 in
  start)
    DAEMON_HOME="`cat /etc/passwd | grep "${DAEMON_ID}" | cut -f6 -d':'`"
    
    #set the current timezone for Java so that log timestamps are accurate
    #we need to use the modern timezone names so that Java can figure out DST
    SYNO_TZ=`cat /etc/synoinfo.conf | grep timezone | cut -f2 -d'"'`
    SYNO_TZ=`grep "^${SYNO_TZ}" /usr/share/zoneinfo/Timezone/tzname | sed -e "s/^.*= //"`
    grep "^export TZ" ${DAEMON_HOME}/.profile > /dev/null \
     && sed -i "s%^export TZ=.*$%export TZ='${SYNO_TZ}'%" ${DAEMON_HOME}/.profile \
     || echo export TZ=\'${SYNO_TZ}\' >> ${DAEMON_HOME}/.profile
    
    su - ${DAEMON_USER} -s /bin/sh -c "${ENGINE_SCRIPT} start ${DAEMON_USER} ${SYNOPKG_PKGDEST} &"
    exit 0
  ;;
  
  stop)
    su - ${DAEMON_USER} -s /bin/sh -c "${ENGINE_SCRIPT} stop ${DAEMON_USER} ${SYNOPKG_PKGDEST}"
    exit 0
  ;;
  
  status)
    if daemon_status ; then
      exit 0
    else
      exit 1
    fi
  ;;
  
  log)
    echo "${SYNOPKG_PKGDEST}/server.log"
    exit 0
  ;;
esac
 

launcher.sh

#!/bin/sh

#--------MINECRAFT/CRAFTBUKKIT server launcher script
#--------package maintained at pcloadletter.co.uk
 
#--------Allows graceful shutdown of server without CPU-specific binaries
#--------You can send commands to the running server like so:
#--------    echo say Hello players >> /tmp/stdin.minecraft
#--------    echo say Hello players >> /tmp/stdin.craftbukkit

DAEMON_USER=$2
SYNOPKG_PKGDEST=$3
DAEMON_USER_SHORT=`echo ${DAEMON_USER} | cut -c 1-8`
JAR_FILE=${SYNOPKG_PKGDEST}/$2.jar

case $1 in
  start)
    if [ -f /tmp/stdin.${DAEMON_USER} ]; then
      rm /tmp/stdin.${DAEMON_USER}
    fi
    touch /tmp/stdin.${DAEMON_USER}
    cd ${SYNOPKG_PKGDEST}
    if [ ! -f syno-marker.txt ]; then
      if [ -f server.properties ]; then
        sed -i "s/A Minecraft Server/A Synology Minecraft Server/" server.properties
  
        #ARM CPU lags a lot, so reduce drawing distance from 10 chunks to 6
        cat /proc/cpuinfo | grep "CPU architecture: 5TE" > /dev/null \
         && sed -i "s/^view-distance=10/view-distance=6/" server.properties
  
        #record that these mods have been made
        echo config updated > syno-marker.txt
      fi
    fi
    JAVA_OPTS='-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalPacing -XX:+AggressiveOpts'
    RAM=$((`free | grep Mem: | sed -e "s/^ *Mem: *\([0-9]*\).*$/\1/"`/1024))
    if [ $RAM -le 128 ]; then
      JAVA_MAX_HEAP=80M
    elif [ $RAM -le 256 ]; then
      JAVA_MAX_HEAP=192M			
    elif [ $RAM -le 512 ]; then
      JAVA_MAX_HEAP=448M
    elif [ $RAM -le 1024 ]; then
      JAVA_MAX_HEAP=896M
    elif [ $RAM -le 2048 ]; then
      JAVA_MAX_HEAP=1792M
    elif [ $RAM -gt 2048 ]; then
      JAVA_MAX_HEAP=2048M
    fi
    JAVA_START_HEAP=${JAVA_MAX_HEAP}
    tail -n 0 -f /tmp/stdin.${DAEMON_USER} | java -Xmx${JAVA_START_HEAP} -Xms${JAVA_MAX_HEAP} ${JAVA_OPTS} -jar ${JAR_FILE} nogui
  ;;

  stop)
    echo say shutting down.. >> /tmp/stdin.${DAEMON_USER}
    sleep 5
    echo stop >> /tmp/stdin.${DAEMON_USER}
    sleep 10
    kill -9 `ps | grep "^ *[0-9]* ${DAEMON_USER_SHORT}.*tail -n 0 -f /tmp/stdin.${DAEMON_USER}" | sed -e "s/^ *\([0-9]*\).*$/\1/"`
    if [ -f /tmp/stdin.${DAEMON_USER} ]; then
      rm /tmp/stdin.${DAEMON_USER}
    fi
  ;;
esac
 

Changelog:

  • 0015 updated to Minecraft 1.5.2, CraftBukkit beta 1.5.2-R0.1
  • 0014 updated to Minecraft 1.4.7, CraftBukkit beta 1.4.7-R0.1, and fixes for DSM 4.2
  • 013 updated to Minecraft 1.4.6, and CraftBukkit beta 1.4.6-R0.3
  • 012 updated to Minecraft 1.4.5, and CraftBukkit 1.3.2-R1.0
  • 011 updated to Minecraft 1.4.2
  • 010 updated to Minecraft 1.3.2, and CraftBukkit 1.3.1-R2.0
  • 009 package scripts fully re-written to unify the Minecraft and CraftBukkit packages
  • 008 updated to Minecraft 1.3.1, unified most scripts into a single installer script, and incorporated minor enhancements from my other packages
  • 007 updated to Minecraft 1.2.3
  • 006 reduced Java max heap to 80MB on 128MB systems, fixed timezone support each server start
  • 005 fixed Java max heap size behaviour on systems with more than 2GB RAM
  • 004 fixed wget SSL problem preventing jar download on some systems
  • 003 updated to Minecraft 1.1
  • 002 server config files are also migrated during version upgrades, and backed up during uninstall. My script’s edits to server.properties are made only once, rather than every startup
  • 001 intial public release
 
 

Unified Windows PE 3.1 builder script for WAIK, with wifi and EFI support

WinPE-with-WLAN

It’s useful to have wifi support with WPA/WPA2 in Windows PE for occasions where you may want to perform an OS install or a salvage on a machine without ethernet built-in: for instance a MacBook Air running Windows or a Sony Vaio P when you don’t have the breakout dongle to hand.

When I started researching wifi support I found that the people who had clearly got it working seemed to be unwilling to share their work. One tool for the job, Holger’s PE Network Manager did not work for me at all, and judging from the dates of the files it had been created for older versions of Windows PE than the current one. There are many third party Windows PE ‘builder’ apps out there that completely replace the official Microsoft WAIK tools. However these seem more aimed at hobbyists than pros and tend to over complicate by adding features that a deployment pro does not really need. Since they’re not-so-trustworthy compiled binaries I don’t feel comfortable using them for work purposes. I decided to carefully work out for myself which registry entries and which additional files are required. There are surprisingly few.

The following unified script will use the standard WAIK tools to create both x86 and x64 builds, or either one individually, and is designed to be run by double-clicking it. Notice the defined variables at the start, particularly the %SOURCE% folder. The script expects the following folders:

  • %SOURCE%\scripts\WinPE – OS build scripts, wifi config XML, optional CA certificates
  • %SOURCE%\drivers\WinPE-x86\CURRENT – drivers
  • %SOURCE%\drivers\WinPE-x64\CURRENT
  • %SOURCE%\tools\WinPE-x86 – optional tools such as GImageX, or apps from portableapps.com
  • %SOURCE%\tools\WinPE-x64
  • %SOURCE%\tools\WinPE-x86\WLAN – the WLAN system files see highlighted section in main script below
  • %SOURCE%\tools\WinPE-x64\WLAN
 

It also requires your WAIK installation to be updated to Windows PE 3.1 using the WAIK supplement for Windows 7 SP1.

I had originally wanted to connect WinPE clients to a WPA-Enterprise network but although Windows PE now supports 802.1x, it appears to work only for wired connections (see my comment on that post).

There is no GUI to manage the WLAN service, but I did find the necessary registry mod to enable the hooks already present in netsh.exe. You will need to use a working Windows 7 PC which is configured for the WLAN of your choice, then export its profile. You cannot connect to a wifi network in Windows PE without having done this. The PSK must not be encrypted since it will be imported onto a different machine (you need to run this with Administrator privileges or else the key parameter will be ignored):

netsh wlan show profiles
netsh wlan export profile name="YOURWLANSSIDHERE" folder="C:\temp" key=clear

Copy this XML file into the folder %SOURCE%\scripts\WinPE in the build tree. When Windows PE boots, you simply type wifi. This wifi.cmd script which is built by the main script will automatically import any XML profiles, list the available wifi networks, and display some example netsh syntax:

netsh wlan connect name=YOURWLANSSIDHERE ssid=YOURWLANSSIDHERE
 

The script

Save as %SOURCE%\scripts\Build_WinPE.cmd

:: Build_WinPE.cmd
::
:: This script will build x86 and x64 Windows PE, automatically
:: collecting drivers from the relevant folders within the
:: unattended installation, building WIM and ISO images, and
:: will also upload the WIM images to the deployment server(s).
::
:: DO NOT cancel this script in progress as you can end up with
:: orphaned locks on files inside mounted WIM images which
:: usually require a reboot of the server to clear.
::

@echo off
setlocal ENABLEDELAYEDEXPANSION

::variables
     set SOURCE=G:\unattended
     set PE_TEMP=C:\temp
     ::WinPE feature pack locale
     set PL=en-US
     ::commma separated list for WDS_SERVERS
     set WDS_SERVERS=FILESERVER1,FILESERVER2
::end variables

if not exist "C:\Program Files\Windows AIK\Tools\PETools\*.*" (
     echo This script requires the Windows Automatic Install Kit to be installed
     echo Download it from http://www.microsoft.com/download/en/details.aspx?id=5753
     echo.
     pause
     goto :eof
)
if "%1"=="relaunch" (
     call :BUILD_WINPE %2 %3 %4
     goto :eof
)
if "%1"=="unmount" (
     :: use this if you have a problem with the script and there are WIMs still mounted
     dism /Unmount-Wim /MountDir:"%PE_TEMP%\WinPE-x86\mount" /discard
     dism /Unmount-Wim /MountDir:"%PE_TEMP%\WinPE-x64\mount" /discard
     goto :eof
)
:prompt
cls
set /P SELECTION=Build WinPE for which CPU architecture (x64, x86, both)? [x64]: 
if "%SELECTION%"=="" set SELECTION=x64
if "%SELECTION%"=="x64" (
     start "Building Windows PE for x64 - NEVER CANCEL THIS SCRIPT IN PROGRESS" cmd /c "%0" relaunch x64 amd64
     goto :eof
)
if "%SELECTION%"=="x86" (
     start "Building Windows PE for x86 - NEVER CANCEL THIS SCRIPT IN PROGRESS" cmd /c "%0" relaunch x86 i386
     goto :eof
)
if "%SELECTION%"=="b" set SELECTION=both
if "%SELECTION%"=="both" (
     ::opening both instances of this script simultaneously seems to cause race conditions with dism.exe
     start /wait "Building Windows PE for x86 - NEVER CANCEL THIS SCRIPT IN PROGRESS" cmd /c "%0" relaunch x86 i386 nopause
     start "Building Windows PE for x64 - NEVER CANCEL THIS SCRIPT IN PROGRESS" cmd /c "%0" relaunch x64 amd64
     goto :eof
)
goto :prompt

:BUILD_WINPE
set PE_ARCH=%1
set PE_ARCH_LONG=%2
echo on
set PATH=%PATH%;C:\Program Files\Windows AIK\Tools\PETools\;C:\Program Files\Windows AIK\Tools\%PROCESSOR_ARCHITECTURE%
set PATH=%PATH%;C:\Program Files\Windows AIK\Tools\Servicing
rd /s /q %PE_TEMP%\WinPE-%PE_ARCH%

::Microsoft haven't used consistent naming in WAIK
if "%PE_ARCH%"=="x86" set WAIK_ARCH=%PE_ARCH%
if "%PE_ARCH%"=="x64" set WAIK_ARCH=%PE_ARCH_LONG%

call copype.cmd %WAIK_ARCH% %PE_TEMP%\WinPE-%PE_ARCH%
::package path
set PP=%ProgramFiles%\Windows AIK\Tools\PETools\%WAIK_ARCH%\WinPE_FPs
::image path
set IP=%PE_TEMP%\WinPE-%PE_ARCH%\mount
echo on
dism /Mount-Wim /WimFile:"%PE_TEMP%\WinPE-%PE_ARCH%\winpe.wim" /Index:1 /MountDir:"%IP%"
dism /image:"%IP%" /Add-Package /PackagePath:"%PP%\winpe-scripting.cab" /PackagePath:"%PP%\%PL%\winpe-scripting_%PL%.cab" /PackagePath:"%PP%\winpe-wmi.cab" /PackagePath:"%PP%\%PL%\winpe-wmi_%PL%.cab" /PackagePath:"%PP%\winpe-mdac.cab" /PackagePath:"%PP%\%PL%\winpe-mdac_%PL%.cab" /PackagePath:"%PP%\WinPE-HTA.cab" /PackagePath:"%PP%\%PL%\WinPE-HTA_%PL%.cab" /PackagePath:"%PP%\WINPE-DOT3SVC.CAB" /PackagePath:"%PP%\%PL%\WINPE-DOT3SVC_%PL%.CAB"
dism /image:"%IP%" /Add-Driver /driver:"%SOURCE%\drivers\WinPE-%PE_ARCH%\CURRENT" /Recurse
copy "%ProgramFiles%\Windows AIK\Tools\PETools\%WAIK_ARCH%\bootsect.exe" "%IP%\Windows"
copy /y "%SOURCE%\scripts\WinPE\*.*" "%IP%\Windows\System32"
copy "%SOURCE%\tools\WinPE-%PE_ARCH%\*.*" "%IP%\Windows\System32"
copy "%ProgramFiles%\Windows AIK\Tools\%WAIK_ARCH%\*.*" "%IP%\Windows\System32"


::add WLAN components
reg load HKLM\PE-BUILD-SYSTEM "%IP%\Windows\System32\config\SYSTEM"
reg load HKLM\PE-BUILD-SOFTWARE "%IP%\Windows\System32\config\SOFTWARE"
set KEY=HKLM\PE-BUILD-SYSTEM\ControlSet001\Enum\Root\LEGACY_WLANSVC
reg add %KEY% /v NextInstance /t REG_DWORD /d 1 /f
reg add %KEY%\0000 /v Service /t REG_SZ /d Wlansvc /f
reg add %KEY%\0000 /v Legacy /t REG_DWORD /d 1 /f
reg add %KEY%\0000 /v ConfigFlags /t REG_DWORD /d 0 /f
reg add %KEY%\0000 /v Class /t REG_SZ /d LegacyDriver /f
reg add %KEY%\0000 /v ClassGUID /t REG_SZ /d {8ECC055D-047F-11D1-A537-0000F8753ED1} /f
reg add %KEY%\0000 /v DeviceDesc /t REG_SZ /d "@%%SystemRoot%%\System32\wlansvc.dll,-257" /f
set KEY=HKLM\PE-BUILD-SYSTEM\ControlSet001\services\wlansvc
reg add %KEY% /v DisplayName /t REG_SZ /d "@%%SystemRoot%%\System32\wlansvc.dll,-257" /f
reg add %KEY% /v ErrorControl /t REG_DWORD /d 1 /f
reg add %KEY% /v Group /t REG_SZ /d TDI /f
reg add %KEY% /v ImagePath /t REG_EXPAND_SZ /d "%%SystemRoot%%\system32\svchost.exe -k LocalSystemNetworkRestricted" /f
reg add %KEY% /v Start /t REG_DWORD /d 2 /f
reg add %KEY% /v Type /t REG_DWORD /d 32 /f
reg add %KEY% /v Description /t REG_SZ /d "@%%SystemRoot%%\System32\wlansvc.dll,-258" /f
reg add %KEY% /v DependOnService /t REG_MULTI_SZ /d "nativewifip\0RpcSs\0Ndisuio\0Eaphost" /f
reg add %KEY% /v ObjectName /t REG_SZ /d LocalSystem /f
reg add %KEY% /v ServiceSidType /t REG_DWORD /d 1 /f
reg add %KEY% /v RequiredPrivileges /t REG_MULTI_SZ /d "SeChangeNotifyPrivilege\0SeImpersonatePrivilege\0SeAuditPrivilege\0SeTcbPrivilege\0SeDebugPrivilege" /f
reg add %KEY% /v FailureActions /t REG_BINARY /d 2c0100000000000000000000030000001400000001000000c0d4010001000000e09304000000000000000000 /f
reg add %KEY%\Enum /v 0 /t REG_SZ /d "Root\LEGACY_WLANSVC\0000" /f
reg add %KEY%\Enum /v Count /t REG_DWORD /d 1 /f
reg add %KEY%\Enum /v NextInstance /t REG_DWORD /d 1 /f
reg add %KEY%\Parameters /v ServiceDll /t REG_EXPAND_SZ /d "%%SystemRoot%%\System32\wlansvc.dll" /f
reg add %KEY%\Parameters /v ServiceDllUnloadOnStop /t REG_DWORD /d 1 /f
reg add %KEY%\Parameters /v ServiceMain /t REG_SZ /d WlanSvcMain /f
reg add HKLM\PE-BUILD-SOFTWARE\Microsoft\NetSh /v wlancfg /t REG_SZ /d wlancfg.dll /f
call :REG_MULTI_SZ-add "HKLM\PE-BUILD-SOFTWARE\Microsoft\Windows NT\CurrentVersion\Svchost" LocalSystemNetworkRestricted wlansvc
xcopy /s "%SOURCE%\tools\WinPE-%PE_ARCH%\WLAN\*.*" "%IP%\Windows"
@echo off
:: These are the files that are required for WLAN (take them from Windows 7 machines of both CPU archs)
:: Put them in %SOURCE%\tools\WinPE-%PE_ARCH%\WLAN and create the same relative folder structure from C:\Windows onwards
:: so the file C:\Windows\inf\netnwifi.inf is copied to %SOURCE%\tools\WinPE-%PE_ARCH%\WLAN\inf\netnwifi.inf
::     C:\Windows\inf\netnwifi.inf
::     C:\Windows\inf\netvwififlt.inf
::     C:\Windows\inf\netvwifimp.inf
::     C:\Windows\l2schemas\wlan_policy_v1.xsd
::     C:\Windows\l2schemas\wlan_profile_v1.xsd
::     C:\Windows\l2schemas\wlanap_profile_v1.xsd
::     C:\Windows\schemas\availablenetwork\availablenetworkinfo.xsd
::     C:\Windows\system32\certutil.exe
::     C:\Windows\system32\wlanapi.dll
::     C:\Windows\system32\wlancfg.dll
::     C:\Windows\system32\wlanhlp.dll
::     C:\Windows\system32\wlanmsm.dll
::     C:\Windows\system32\wlansec.dll
::     C:\Windows\system32\wlansvc.dll
::     C:\Windows\system32\wlanui.dll
::     C:\Windows\system32\wlgpclnt.dll
::     C:\Windows\system32\drivers\nwifi.sys
::     C:\Windows\system32\drivers\vwififlt.sys
::     C:\Windows\system32\drivers\vwifimp.sys
::     C:\Windows\system32\en-US\certutil.exe.mui (or your locale's equivalent)
::     C:\Windows\system32\en-us\wlanapi.dll.mui
::     C:\Windows\system32\en-us\wlancfg.dll.mui
::     C:\Windows\system32\en-us\wlansvc.dll.mui
::     C:\Windows\system32\en-us\wlanui.dll.mui
::     C:\Windows\system32\en-us\wlgpclnt.dll.mui
echo on
reg unload HKLM\PE-BUILD-SYSTEM
reg unload HKLM\PE-BUILD-SOFTWARE
:: build wifi.cmd
(
     echo @echo off
     echo drvload X:\WINDOWS\Inf\netvwifimp.inf
     echo drvload X:\WINDOWS\Inf\netvwififlt.inf
     echo drvload X:\WINDOWS\Inf\netnwifi.inf
     echo netcfg -c s -i ms_nativewifip
     echo echo Importing detected CA certificate^(s^).
     echo for %%%%i in ^(*.cer^) do certutil -addstore root %%%%i
     echo echo.
     echo net start dot3svc
     echo net start wlansvc
     echo for %%%%i in ^("Wireless Network Connection*.xml"^) do netsh wlan add profile filename="%%%%i"
     echo netsh wlan show networks
     echo echo.
     echo echo use "netsh wlan" to manage the wifi connection like so:
     echo echo   netsh wlan connect name=YOURWLANHERE ssid=YOURWLANHERE
     echo echo   ipconfig /renew
     echo echo.
) > "%IP%\Windows\System32\wifi.cmd"

dism /Unmount-Wim /MountDir:"%IP%" /commit
imagex /export /boot /compress fast "%PE_TEMP%\WinPE-%PE_ARCH%\winpe.wim" 1 "%PE_TEMP%\WinPE-%PE_ARCH%\ISO\sources\boot.wim"
@echo off

::Mac OS BootCamp will look for autorun.inf in order to validate this disk as a Windows Installer CD
::adding this allows us to start unattended installs using WinPE
date /T > "%PE_TEMP%\WinPE-%PE_ARCH%\ISO\autorun.inf"

::x64 bootable ISO includes both BIOS & EFI boot loaders
if "%PE_ARCH%" == "x64" (
     set CD_CMD=oscdimg -m -o -u2 -udfver102 -bootdata:2#p0,e,b"%PE_TEMP%\WinPE-%PE_ARCH%\etfsboot.com"#pEF,e,b"%PE_TEMP%\WinPE-%PE_ARCH%\efisys.bin" "%PE_TEMP%\WinPE-%PE_ARCH%\ISO" "%PE_TEMP%\WinPE-%PE_ARCH%\winpe_%PE_ARCH_LONG%.iso"
) else ( 
     set CD_CMD=oscdimg -n -b"%PE_TEMP%\WinPE-%PE_ARCH%\etfsboot.com" "%PE_TEMP%\WinPE-%PE_ARCH%\ISO" "%PE_TEMP%\WinPE-%PE_ARCH%\winpe_%PE_ARCH_LONG%.iso"
)
echo on
%CD_CMD%
@echo off

::rename the WIM file to avoid having multiple image files on the WDS server with the same filename
ren "%PE_TEMP%\WinPE-%PE_ARCH%\ISO\sources\boot.wim" boot_%PE_ARCH_LONG%.wim
del "%PE_TEMP%\WinPE-%PE_ARCH%\winpe.wim"

for %%i in (%WDS_SERVERS%) do (
     echo.
     echo Adding/updating boot image on WDS server: %%i
     :: try to add the image first, if that fails then replace existing
     wdsutil /Verbose /Progress /Add-Image /ImageFile:"%PE_TEMP%\WinPE-%PE_ARCH%\ISO\sources\boot_%PE_ARCH_LONG%.wim" /Server:%%i /ImageType:Boot /Name:"Microsoft Windows PE (%PE_ARCH%)" || wdsutil /Verbose /Progress /Replace-Image /Image:"Microsoft Windows PE (%PE_ARCH%)" /ImageType:Boot /Architecture:%PE_ARCH% /ReplacementImage /Name:"Microsoft Windows PE (%PE_ARCH%)" /ImageFile:"%PE_TEMP%\WinPE-%PE_ARCH%\ISO\sources\boot_%PE_ARCH_LONG%.wim" /Server:%%i
     echo.
)
::rename the WIM back again so bootable USB devices can be created
ren "%PE_TEMP%\WinPE-%PE_ARCH%\ISO\sources\boot_%PE_ARCH_LONG%.wim" boot.wim
echo *******************************************************************
echo WDS boot image(s) updated
echo.
echo A bootable ISO of this image has been created at:
echo   %PE_TEMP%\WinPE-%PE_ARCH%\winpe_%PE_ARCH_LONG%.iso
echo.
echo To create a bootable USB key, use diskpart.exe to create a FAT32 partition
echo and mark it active, then copy the contents of this folder to its root:
echo   %PE_TEMP%\WinPE-%PE_ARCH%\ISO
echo.
echo FAT32 is required for EFI support.
echo.
if "%3"=="nopause" goto :eof
pause
goto :eof

:REG_MULTI_SZ-add
::subroutine to append a value to a multiple string value Registry entry
setlocal ENABLEEXTENSIONS
set KEY=%1
set VALUE=%2
for /f "tokens=2*" %%a in ('reg query %KEY% /v %VALUE% /t REG_MULTI_SZ ^| FIND "REG_MULTI_SZ"') do set DATA=%%b
set DATA=%DATA%\0%3
reg add %KEY% /v %VALUE% /t REG_MULTI_SZ /d %DATA% /f
 
 

Java SE Embedded package for Synology NAS

UPDATE 4 – Now supports DS414j!
UPDATE 3 – Now with Java 8, and full support for all versions on Intel CPUs running DSM 5.0!
UPDATE 2 – Now running on PowerPC 2010 Synology models running DSM 5.0 beta!
UPDATE – Now running on Armada 370/XP (ARMv7) and Intel Atom Evansport CPUs, as used in some x14 series Synology models!

There is a lot of good Java software, but it’s difficult to get Java running on a NAS. I decided to do something about that, mainly to pave the way for a Serviio package that didn’t require a huge guide to install. To stay legal, I followed how QNAP packaged Java for their NAS products. The Java binaries cannot be redistributed so this package looks for the downloaded .tar.gz archive in the public shared folder on the NAS. The end user is required to register with Oracle to get the file, as you can see from the description below. I also added locale support to the underlying Linux since Java requires this for UTF-8 support when reading from the filesystem.

Only Marvell Kirkwood, Marvell Armada 370/XP, Intel, and Freescale QorIQ/PowerQUICC PowerPC CPUs are supported, so please check which CPU your NAS has. Java 7 for Intel Atom CPU does not currently work on Synology systems running DSM versions older than 5.0, so those users are limited to Java 6. Though Oracle have published a JRE for PowerPC, it also requires DSM 5.0. PowerQUICC PowerPC synos running older DSM versions do not have a recent enough glibc version (2.3.4, but Oracle’s Java needs 2.4). It is technically possible to run Java on those units, but it requires chroot-ing to a Debian install which can get complicated. Christophe from chreggy.fr has recently released packages to automate this.

Java package for Synology

 

Instructions

  • If you have a multi-bay NAS, use the Shared Folder control panel to create a shared folder called public (it must be all lower case). On single bay models this is created by default. Assign it with Read/Write privileges for everyone.
  • Register with Oracle and download Java SE for Embedded 7 for either ARMv5, ARMv7 Headless Server, or PowerPC e500v2 as appropriate (Intel NAS users will need the end-of-life Java SE for Embedded 6), and save in the public shared folder. If the file isn’t found an error message will display the full expected filename. You cannot use a different version by renaming it, since the folder structure inside the archive will be different.
  • Install the Java package from the DSM Package Center. In Settings -> Package Sources add my package repository URL which is http://packages.pcloadletter.co.uk. My packages will appear in the Community section once the repo URL is added.
  • If you have trouble getting the Java archive recognised, try downloading it with a different web browser. Some browsers try to help by uncompressing the file, or renaming it without warning. I have tried to code around most of these behaviours.
  • The package does not need to start, just installing it is sufficient.
  • If you update DSM later, you will need to re-install this package or else UTF-8 and locale support will be broken by the update.
 

Package scripts

For information, here are the package scripts so you can see what it’s going to do. You can learn more about how Synology packages work by reading the Synology Package wiki.

installer.sh

#!/bin/sh

#--------JAVA installer script
#--------package maintained at pcloadletter.co.uk

COMMENT="# Synology Java Package"
SYNO_CPU_ARCH="`uname -m`"
[ "`echo ${SYNO_CPU_ARCH} | cut -c1-7`" == "armv5te" ] && SYNO_CPU_ARCH="armv5tel"
cat /proc/cpuinfo | grep "Comcerto 2000" > /dev/null && SYNO_CPU_ARCH="armv7l-neon"
[ "${SYNO_CPU_ARCH}" == "x86_64" ] && SYNO_CPU_ARCH="i686"

if [ "${SYNOPKG_PKGNAME}" == "java6" ]; then
  DOWNLOAD_URL="http://tinyurl.com/javaembedarchive"
  EXTRACTED_FOLDER="ejre1.6.0_38"
  if [ "`echo ${SYNO_CPU_ARCH} | cut -c1-7`" == "armv5te" ]; then
    JAVA_BINARY="ejre-1_6_0_38-fcs-b05-linux-arm-sflt-eabi-headless-13_nov_2012.tar.gz"
    JAVA_BUILD="ARMv5 Linux - Headless EABI"
  elif [ "${SYNO_CPU_ARCH}" == "armv7l" ]; then
    JAVA_BINARY="ejre-1_6_0_38-fcs-b05-linux-arm-vfp-eabi-headless-13_nov_2012.tar.gz"
    JAVA_BUILD="ARMv6/7 Linux - Headless EABI, VFP, SoftFP ABI, Little Endian"
  elif [ "${SYNO_CPU_ARCH}" == "i686" ]; then
    JAVA_BINARY="ejre-1_6_0_38-fcs-b05-linux-i586-headless-13_nov_2012.tar.gz"
    JAVA_BUILD="x86 Linux Small Footprint - Headless"
  elif [ "${SYNO_CPU_ARCH}" == "ppc" ]; then
    JAVA_BINARY="ejre-1_6_0_38-fcs-b05-linux-ppc-e500v2-headless-13_nov_2012.tar.gz"
    JAVA_BUILD="Power Architecture Linux - Headless - e500v2 core"
  fi    

elif [ "${SYNOPKG_PKGNAME}" == "java7" ]; then
  DOWNLOAD_URL="http://tinyurl.com/javaembed"
  EXTRACTED_FOLDER="ejre1.7.0_60"
  if [ "${SYNO_CPU_ARCH}" == "armv5tel" ]; then
    JAVA_BINARY="ejre-7u60-fcs-b19-linux-arm-sflt-headless-07_may_2014.tar.gz"
    JAVA_BUILD="ARMv5 Linux - Headless EABI, SoftFP ABI, Little Endian"
  elif [ "${SYNO_CPU_ARCH}" == "armv7l" ]; then
    JAVA_BINARY="ejre-7u60-fcs-b19-linux-arm-vfp-sflt-client_headless-07_may_2014.tar.gz"
    JAVA_BUILD="ARMv6/7 Linux - Headless - Client Compiler EABI, VFP, SoftFP ABI, Little Endian"
  elif [ "${SYNO_CPU_ARCH}" == "armv7l-neon" ]; then
    JAVA_BINARY="ejre-7u60-fcs-b19-linux-arm-vfp-hflt-client_headless-07_may_2014.tar.gz"
    JAVA_BUILD="ARMv6/7 Linux - Headless - Client Compiler EABI, VFP, HardFP ABI, Little Endian"
  elif [ "${SYNO_CPU_ARCH}" == "i686" ]; then
    JAVA_BINARY="ejre-7u60-fcs-b19-linux-i586-headless-07_may_2014.tar.gz"
    JAVA_BUILD="x86 Linux Small Footprint - Headless"  
  elif [ "${SYNO_CPU_ARCH}" == "ppc" ]; then
    JAVA_BINARY="ejre-7u60-fcs-b19-linux-ppc-e500v2-headless-07_may_2014.tar.gz"
    JAVA_BUILD="Power Architecture Linux - Headless - e500v2 with double-precision SPE Floating Point Unit"
  fi

elif [ "${SYNOPKG_PKGNAME}" == "java8" ]; then
  DOWNLOAD_URL="http://tinyurl.com/javaembed"
  EXTRACTED_FOLDER="ejdk1.8.0_06"
  if [ "${SYNO_CPU_ARCH}" == "armv5tel" ]; then
    JAVA_BINARY="ejdk-8u6-fcs-b23-linux-arm-sflt-12_jun_2014.tar.gz"
    JAVA_BUILD="ARMv5 Linux - Headless EABI, SoftFP ABI, Little Endian"
  elif [ "${SYNO_CPU_ARCH}" == "armv7l" ]; then
    JAVA_BINARY="ejdk-8u6-fcs-b23-linux-arm-vfp-sflt-12_jun_2014.tar.gz"
    JAVA_BUILD="ARMv6/7 Linux - VFP, SoftFP ABI, Little Endian"
  elif [ "${SYNO_CPU_ARCH}" == "armv7l-neon" ]; then
    JAVA_BINARY="ejdk-8u6-fcs-b23-linux-arm-vfp-hflt-12_jun_2014.tar.gz"
    JAVA_BUILD="ARMv7 Linux - VFP, HardFP ABI, Little Endian"
  elif [ "${SYNO_CPU_ARCH}" == "i686" ]; then
    JAVA_BINARY="ejdk-8u6-fcs-b23-linux-i586-12_jun_2014.tar.gz"
    JAVA_BUILD="x86 Linux Small Footprint - Headless"
  elif [ "${SYNO_CPU_ARCH}" == "ppc" ]; then
    JAVA_BINARY="ejdk-8u6-fcs-b23-linux-ppc-e500v2-12_jun_2014.tar.gz"
    JAVA_BUILD="Power Architecture Linux - Headless - e500v2 with double-precision SPE Floating Point Unit"
  fi
fi

JAVA_BINARY=`echo ${JAVA_BINARY} | cut -f1 -d'.'`
PUBLIC_FOLDER="`cat /usr/syno/etc/smb.conf | sed -r '/\/public$/!d;s/^.*path=(\/volume[0-9]{1,4}\/public).*$/\1/'`"
TEMP_FOLDER="`find / -maxdepth 2 -name '@tmp' | head -n 1`"
NATIVE_BINS_URL="http://packages.pcloadletter.co.uk/downloads/java-native-${SYNO_CPU_ARCH}.tgz"   
NATIVE_BINS_FILE="`echo ${NATIVE_BINS_URL} | sed -r "s%^.*/(.*)%\1%"`"
#DSM versions older than 4.3 need locale support adding, don't download unless needed
[ ! -e /usr/bin/locale ] && INSTALL_FILES="${NATIVE_BINS_URL}"
source /etc/profile


preinst ()
{
  if [ -z ${PUBLIC_FOLDER} ]; then
    echo "A shared folder called 'public' could not be found - note this name is case-sensitive. "
    echo "Please create this using the Shared Folder DSM Control Panel and try again."
    exit 1
  fi

  JAVA_BINARY_FOUND=

  if [ ! -z ${JAVA_HOME} ]; then
    echo "It seems from /etc/profile that a Java Runtime is already installed at ${JAVA_HOME}. Uninstall it and try again."
    exit 1
  fi
  
  [ -f ${PUBLIC_FOLDER}/${JAVA_BINARY}.tar.gz ] && JAVA_BINARY_FOUND=true
  [ -f ${PUBLIC_FOLDER}/${JAVA_BINARY}.tar ] && JAVA_BINARY_FOUND=true
  [ -f ${PUBLIC_FOLDER}/${JAVA_BINARY}.tar.tar ] && JAVA_BINARY_FOUND=true
  [ -f ${PUBLIC_FOLDER}/${JAVA_BINARY}.gz ] && JAVA_BINARY_FOUND=true
  
  if [ -z ${JAVA_BINARY_FOUND} ]; then
    echo "Java binary bundle not found. " > $SYNOPKG_TEMP_LOGFILE
    echo "I was expecting the file ${PUBLIC_FOLDER}/${JAVA_BINARY}.tar.gz. "
    echo "Please agree to the Oracle licence at ${DOWNLOAD_URL}, then download the '${JAVA_BUILD}' package"
    echo "and place it in the 'public' shared folder on your NAS. This download cannot be automated even if "
    echo "displaying a package EULA could potentially cover the legal aspect, because files hosted on Oracle's "
    echo "server are protected by a session cookie requiring a JavaScript enabled browser."
    exit 1
  fi

  cd ${TEMP_FOLDER}
  for WGET_URL in ${INSTALL_FILES}
  do
    WGET_FILENAME="`echo ${WGET_URL} | sed -r "s%^.*/(.*)%\1%"`"
    [ -f ${TEMP_FOLDER}/${WGET_FILENAME} ] && rm ${TEMP_FOLDER}/${WGET_FILENAME}
    wget ${WGET_URL}
    if [[ $? != 0 ]]; then
      if [ -d ${PUBLIC_FOLDER} ] && [ -f ${PUBLIC_FOLDER}/${WGET_FILENAME} ]; then
        cp ${PUBLIC_FOLDER}/${WGET_FILENAME} ${TEMP_FOLDER}
      else     
        echo "There was a problem downloading ${WGET_FILENAME} from the official download link, "
        echo "which was \"${WGET_URL}\" "
        echo "Alternatively, you may download this file manually and place it in the 'public' shared folder. "
        exit 1
      fi
    fi
  done
  
  exit 0
}


postinst ()
{
  WORKING_DIR=`pwd`
  
  #ldd tool is very useful but not included in DSM
  [ ! -e /bin/ldd ] && cp ${SYNOPKG_PKGDEST}/tools/ldd-${SYNO_CPU_ARCH} /bin/ldd

  #DSM versions older than 4.3 need locale support adding
  if [ ! -e /usr/bin/locale ]; then
    #extract native binaries
    cd ${SYNOPKG_PKGDEST}
    tar xzf ${TEMP_FOLDER}/${NATIVE_BINS_FILE} && rm ${TEMP_FOLDER}/${NATIVE_BINS_FILE}

    #build missing locale with UTF-8 support (don't think it matters which language)
    cp ${SYNOPKG_PKGDEST}/bin/* /bin
    cp -R ${SYNOPKG_PKGDEST}/share/i18n /usr/share
    if [ ! -d /usr/lib/locale ]; then
      mkdir /usr/lib/locale
    fi
    localedef -c -f UTF-8 -i en_US en_US.utf8
  fi  

  #extract Java (Web browsers love to interfere with .tar.gz files)
  cd ${PUBLIC_FOLDER}
  if [ -f ${JAVA_BINARY}.tar.gz ]; then
    #Firefox seems to be the only browser that leaves it alone
    tar xzf ${JAVA_BINARY}.tar.gz
  elif [ -f ${JAVA_BINARY}.gz ]; then
    #Chrome
    tar xzf ${JAVA_BINARY}.gz
  elif [ -f ${JAVA_BINARY}.tar ]; then
    #Safari
    tar xf ${JAVA_BINARY}.tar
  elif [ -f ${JAVA_BINARY}.tar.tar ]; then
    #Internet Explorer
    tar xzf ${JAVA_BINARY}.tar.tar
  fi
  
  #install Java
  if [ "${EXTRACTED_FOLDER}" != "${EXTRACTED_FOLDER/jdk/}" ]; then
    mv ${EXTRACTED_FOLDER} ${SYNOPKG_PKGDEST}
    JRE_PATH="`find ${SYNOPKG_PKGDEST}/${EXTRACTED_FOLDER} -name jre`"
  else
    mv ${EXTRACTED_FOLDER} ${SYNOPKG_PKGDEST}/jre
    JRE_PATH="${SYNOPKG_PKGDEST}/jre"
  fi
  
  #change owner of folder tree
  chown -R root:root ${SYNOPKG_PKGDEST}

  echo "NOTE - This package does *not* start and stop like other packages. "
  echo "Java is correctly installed if you can see the runtime and HotSpot version numbers, "
  echo "and locale information in the package Log tab."
  
  exit 0
}


preuninst ()
{
  exit 0
}


postuninst ()
{
  #clean up profile mods
  sed -i "/${COMMENT}/d" /etc/profile
  sed -i "/${COMMENT}/d" /root/.profile

  #leave locale support in place on older DSM versions - too risky to delete system binaries
  
  exit 0
}
 

start-stop-status.sh

#!/bin/sh

JRE_PATH="`find ${SYNOPKG_PKGDEST} -name jre`"
COMMENT="# Synology Java Package"

EnvCheck()
#updates to DSM will reset these changes so check them each startup 
{
  #/etc/profile should contain 5 lines added by this package tagged with trailing comments
  COUNT=`grep -c "$COMMENT$" /etc/profile`
  if [ $COUNT != 5 ]; then

    #remove any existing mods
    sed -i "/${COMMENT}/d" /etc/profile

    #add required environment variables
    echo "PATH=\$PATH:${JRE_PATH}/bin ${COMMENT}" >> /etc/profile
    echo "JAVA_HOME=${JRE_PATH} ${COMMENT}" >> /etc/profile
    echo "CLASSPATH=.:${JRE_PATH}/lib ${COMMENT}" >> /etc/profile
    echo "LANG=en_US.utf8 ${COMMENT}" >> /etc/profile
    echo "export CLASSPATH JAVA_HOME LANG PATH ${COMMENT}" >> /etc/profile
  fi

  #/root/.profile should contain 3 lines added by this package tagged with trailing comments
  COUNT=`grep -c "$COMMENT$" /root/.profile`
  if [ $COUNT != 3 ]; then

    #remove any existing mods
    sed -i "/${COMMENT}/d" /root/.profile

    #add required environment variables
    echo "PATH=\$PATH:${JRE_PATH}/bin ${COMMENT}" >> /root/.profile

    #set the current timezone for Java so that log timestamps are accurate
    #we need to use the modern timezone names so that Java can figure out DST
    #this should be configured in each Java daemon's start script to that changes
    #to timezone settings in DSM are reflected
    #we need to use the modern timezone names so that Java can figure out DST

    SYNO_TZ=`cat /etc/synoinfo.conf | grep timezone | cut -f2 -d'"'`
    SYNO_TZ=`grep "^${SYNO_TZ}" /usr/share/zoneinfo/Timezone/tzname | sed -e "s/^.*= //"`
    echo TZ=\'${SYNO_TZ}\' ${COMMENT} >> /root/.profile
    echo "export PATH TZ ${COMMENT}" >> /root/.profile
  fi
}

case $1 in
  start)
    EnvCheck
    source /etc/profile
    source /root/.profile

    #evidence of whether Java can start successfully is written to the package log
    java -version > ${SYNOPKG_PKGDEST}/output.log 2>&1
    echo >> ${SYNOPKG_PKGDEST}/output.log
    echo System installed locales: >> ${SYNOPKG_PKGDEST}/output.log
    locale -a >> ${SYNOPKG_PKGDEST}/output.log
    echo >> ${SYNOPKG_PKGDEST}/output.log
    echo JAVA_HOME=$JAVA_HOME >> ${SYNOPKG_PKGDEST}/output.log
    echo TZ=$TZ >> ${SYNOPKG_PKGDEST}/output.log

    exit 0
  ;;

  stop)
    exit 0
  ;;

  status)
    source /etc/profile
    source /root/.profile
    if [ -e "${JAVA_HOME}/bin/java" ]; then
      exit 0
    else
      exit 1
    fi
  ;;

  log)
    echo "${SYNOPKG_PKGDEST}/output.log"
    exit 0
  ;;

esac
 

Changelog:

  • 0027 Updated Java 8 to 8u6m, added support for Mindspeed Comcerto 2000 CPU in DS414j for Java 7 and Java 8
  • 0026 DSM updates will no longer break Java
  • 0025 Updated to Java 7u60
  • 0024 Updated to Java 7u55
  • 0023 Added Java 8 support
  • 0022 Updated to Java 7u51
  • 0021 Locale support is no longer downloaded and added to DSM 4.3 or newer since it’s already present
  • 0020 Added support for Intel Atom Evansport and Armada XP CPUs in new DSx14 products, removed Early Access JDK 8 support since it did not run on the Armada SoCs
  • 0019 Updated to Java 7u45
  • 0018 Updated to Java 7u40, and JDK8 Early Access b106
  • 0017 For Armada370 CPU changed from ARMv7 Server JVM to Client JVM since the Server one turned out to be unstable. Added support for Early Access JDK 8 for Armada 370.
  • 0016 Added Armada370 CPU support (DS213j)
  • 0015 Updated to Java 7u21, Java 6 Embedded seems to be no longer maintained by Oracle
  • 0014 Fixed metadata for DSM 4.2 Package Center
  • 013 Updated to Java 6u38 and Java 7u10
  • 012 Installer script fetches native binaries separately to reduce bandwidth on repo hosting
  • 011 Added support for Freescale QorIQ PowerPC CPUs used in some Synology x13 series products, PowerQUICC PowerPC processors in previous Synology generations with older glibc versions are not supported
  • 010 updated to Java 6u34 and Java 7u6
  • 009 corrected timezone bug for Atlantic and Pacific timezones which may have caused error messages when in fact the package had installed correctly
  • 008 unified the build scripts
  • 007 included the locale binaries to simplify installation, fixed environment variables for root user (inc. timezone with DST support), displayed Java version information and env vars in Log tab, incremented Java to latest versions 6u32 and 7u4, fixed detection of renamed .tar.gz archives downloaded by Google Chrome
  • 006 fixed path issue for root user on non-bootstrapped systems, updated info link to refer back to this page, and improved description text
  • 005 incremented JRE versions to match new releases from Oracle
  • 004 forced check for existence of ‘public’ shared folder
  • 003 fixed a stupid typo that stopped 002 from working :)
  • 002 rewrote scripts to be easier to maintain, and to allow for web browsers that untar or rename the Java and toolchain binary bundles as they’re saved
  • 001 fixed package repo support, defined as a non-runnable service, prevented more than one JRE being installed, saved files into actual package folder rather than @appstore/java now that package names can be separate from Package Center display names
  • v3 initial spk release (I think!)
 
 

Batch script for recursive FFmpeg transcoding

I recently had to shrink around 50GB of MP3 audio recordings that were sitting in a nested folder structure on a web server. Having experimented to find more appropriate LAME encoder settings for spoken word content I needed to transcode the files whilst keeping the existing ID3 tags intact. FFmpeg can do this nicely using libmp3lame, whereas LAME by itself cannot. Armed with my own compile of FFmpeg, I created a drag & drop batch script to recursively work through a folder structure transcoding MP3 and WAV files and writing out the resulting MP3 files in the same folder structure with an amended top level folder name. It will accept multiple files or folders being dragged and dropped. You could adapt this script to whatever task you’re using FFmpeg for.

::MP3 transcoding to more sensible default quality settings for spoken word
::32kHz mono VBR @ approx. 64Kbps
::
::pcloadletter.co.uk

@echo off
setlocal ENABLEDELAYEDEXPANSION
::parse multiple command lines (so multiple targets can be dragged and dropped at once)
:commandlineloop
if "%~1"=="" goto :continue
call :mainloop %1
shift
goto :commandlineloop
:continue
echo.
pause
goto :eof

::main loop we run for each command line arg of the script
:mainloop
set folderpath=
if "%~1"=="" (
  echo Drag and drop files or folders onto this script.
  goto :eof
)
set folderpath=%~1
echo "%1" | find /I ".mp3" && (
  goto :fileonly
)
echo "%1" | find /I ".wav" && (
  goto :fileonly
)

::folders
echo.
echo.
echo Processing folder %1
echo ________________________________________________________________________________
echo.

::we need to find the last folder level in the folderpath
::batch scripting is horrible so we need to use a hacky way to get the last token
::(from http://stackoverflow.com/questions/5473840/last-token-in-batch-variable)
set temp_string=%~1
set count=0

::iterate parsing the tokens and trimming the string until it's empty, while counting how many loops
:loopcounter
for /F "tokens=1* delims=\" %%a in ( "%temp_string%" ) do (
  set /A count+=1
  set temp_string=%%b
  goto loopcounter
)
for /F "tokens=%count% delims=\" %%i in ( "%~1" ) do set folder=%%i

::append the top level folder name with "-optimized" so we create a new folder tree
for /R "%folderpath%" %%i in (*.mp3 *.wav) do (
  set sourcepath=%%~di%%~pi
  set destpath=!sourcepath:%folder%=%folder%-optimized!
  md "!destpath!"
  ffmpeg -i "%%~fi" -acodec libmp3lame -aq 8 -ar 32000 -ac 1 "!destpath!%%~ni.mp3"
)
goto :eof

::individual files
:fileonly
for %%i in ("%folderpath%") do (
  ffmpeg -i "%folderpath%" -acodec libmp3lame -aq 8 -ar 32000 -ac 1 "%%~di%%~pi%%~ni-optimized.mp3"
)
 

There are several neat little tricks here. Multiple folders can be dragged onto the script because it uses SHIFT to work through each command line argument. The next hurdle is to get the last part of the folder name(s) being dragged. Though this would be simple in any Unix-like shell, but to do this in batch without relying on any additional tools proved to be quite tricky. Though tokens can be parsed by FOR, finding the last one (which we need) requires us to count how many tokens there are. Only then can the last one be selected. The folder name string substitution is done by the SET command (also used in my file renaming script). Delayed variable expansion means that the variables between the the exclamation marks are evaluated once per loop rather than the default of once per script execution (more info here). I hadn’t realised until today that you can create a folder structure several layers deep with a single MD command. This avoids having to iterate through all subfolders – we can use FOR’s /R switch to handle the recursion in a single line. For more information on some of the variables containing tildes like %%~pi and %%~ni, try running FOR /?. The script also first runs the FFmpeg subroutine with ECHO in front of the commands so you can double-check the syntax before proceeding.

There are of course much more efficient ways of compressing spoken word content than MP3 these days – AAC-HEv2 for instance, but that will rule out all but the latest audio playback devices.
The source MP3 recordings were in 44.1kHz mono @ 128Kbps. This was definitely overkill for speech. Just by opting for 64Kbps I could halve that. However I couldn’t drop the sample rate below 32kHz, as this is the lowest legal limit of MPEG-1 layer III. This info is tricky to find but the LAME encoder reveals it in its extended help:

MPEG-1   layer III sample frequencies (kHz):  32  48  44.1
bitrates (kbps): 32 40 48 56 64 80 96 112 128 160 192 224 256 320
MPEG-2   layer III sample frequencies (kHz):  16  24  22.05
bitrates (kbps):  8 16 24 32 40 48 56 64 80 96 112 128 144 160
MPEG-2.5 layer III sample frequencies (kHz):   8  12  11.025
bitrates (kbps):  8 16 24 32 40 48 56 64

 

I think it’s safe to assume that most MP3 players can handle variable bit rate. As far as I can remember only the first generation ones couldnt from early last decade – the sort of ones which only had around 64MB of storage. I don’t think excluding players this old is much of a concern.

Playing around with VBR quality settings in LAME we can see that with -V 9 we get a small file (12.9MB becomes 3.7MB) however it’s actually an MPEG-2 layer III file which as you can see above supports lower sample rates. I’m not certain of the wider compatibility of this sub-type of MP3 file, and besides it does sound considerably worse than the lowest MPEG-1 layer III settings which is –V 8 (12.9MB becomes 5.2MB). This proved to be the best compromise between size and quality.

Windows 7 new user profile – MSOE.DLL could not be loaded error

Since my Windows 7 deployment I get an error every time a new user logs in (or logs in following a profile reset).

Windows Mail could not be started because MSOE.DLL could not be loaded

Since you can click past it I wasn’t initially too concerned but I soon came to realize that there’s pretty much no information on the Web about this problem. Anything the search engines turn up is invariably about Outlook Express on Windows 98/Me.

It seems to be caused when some kind of IE First Run process opens Internet Explorer with an MSN page and presumably tries to register the default email client (which is not yet set to Outlook, despite it being installed). The strange thing is that Windows Mail is not even available on Windows 7. Microsoft discontinued it after Vista in favour of Windows Live Mail. WinMail.exe is present on the system but it’s hidden, and that MSOE.DLL has been purposely omitted. Apparently you can get it working again by supplying a copy from a Vista PC.

The problem is that this error looks important enough for a new user to contact the helpdesk about. The other issue is that it significantly delays that first logon, sometimes by several entire minutes. The error occurs immediately after you enter your credentials (you can hear the alert sound) but it happens while the desktop is still hidden from view by the Windows 7 splash screen. Eventually the Preparing your desktop splash screen times out and you can click OK:

Windows Mail could not be started because MSOE.DLL could not be loaded

Notice the label in that floating window – it’s setting up a component which is titled as Microsoft Windows.

Eventually I discovered a KB article relating to Internet Explorer 5 personalized settings, which was when Microsoft introduced this method of installation.

Having consulted that document I viewed HKLM\SOFTWARE\Microsoft\ActiveSetup\InstalledComponents in Regedit and saw that there is indeed a component with the name “Microsoft Windows” which is invoking WinMail.exe, intending to register it as a mail and news (nntp) client:

WinMail.exe ActiveSetup parameters

I have no idea why it’s doing this since WinMail.exe is intentionally disabled in Windows 7. It would seem logical that deleting this whole key should fix the problem (I backed it up first of course). In testing this didn’t seem to work though.

However I did notice that sometimes I got the error on screen twice and other times only once. Exasperated I searched the registry for instances of WinMail.exe until I discovered the reason: the entire ActiveSetup branch of the registry also exists under HKLM\Software\WOW6432Node\Microsoft, for the 32bit version of Internet Explorer on 64bit systems. Once both keys had been deleted the error disappeared.

I was then able to refine this – instead of deleting the whole key you only need to set to the single DWORD value IsInstalled to zero. Do this to both instances and the problem is fixed! In my environment I used my existing VBScript startup script which worktations inherit by Group Policy. Excerpt below:

If InStr (strOS,"Microsoft Windows 7") Then
  'Prevent Windows from trying to invoke the non-existent Windows Mail for new user profile setup
  On Error Resume Next
  strKey = "SOFTWARE\Microsoft\Active Setup\Installed Components\{44BBA840-CC51-11CF-AAFA-00AA00B6015C}"
  objReg.SetDWORDValue HKEY_LOCAL_MACHINE,strKey,"IsInstalled",0
  strKey = "SOFTWARE\Wow6432Node\Microsoft\Active Setup\Installed Components\{44BBA840-CC51-11CF-AAFA-00AA00B6015C}"
  objReg.SetDWORDValue HKEY_LOCAL_MACHINE,strKey,"IsInstalled",0
  On Error Goto 0

Enabling Network Level Authentication on Windows XP by script

Migrating to Windows 7 has thrown up another problem – users wanting to connect from home computers running XP cannot use the Remote Desktop Client to connect to their newly upgraded office PCs. The Network Level Authentication change to the Remote Desktop Client was made because the original RDP is susceptible to Man-in-the-middle attacks.

Rather than leaving the new systems vulnerable by allowing connections from all clients in Computer Propertes > Remote settings, I discovered that Windows XP SP3 does in fact offer NLA support however it’s disabled by default. Somewhat frustratingly, the steps outlined in Microsoft KB 951608 require Registry edits which I would not want to encourage non-IT-savvy people to try. Giving out a .reg file is not really a good idea here either since these are additions to existing values, so forced replacements could interfere with certain vendors’ VPN clients etc.

Here’s a VBScript for the task which will only install on XP SP3 and will detect if the modifications have already been made. You could easily target it at a whole group of PCs by iterating through an array of hostnames.

'Enables Network Level Authentication on XP SP3 (disabled by default)
'which allows you to use the Remote Desktop Client 6.1 to connect to
'Windows 7 and Windows Server 2008 R2 without degrading security

Option Explicit

Const HKEY_LOCAL_MACHINE = &H80000002

Dim strLsaKey, strLsaValue, strHostname, size, arrMultiRegSZ, objReg, objWMI, colItems, i, found, modified
Dim objItem, SPlevel, strOSVer, strSecProvKey, strSecProvValue, strValue

strLsaKey = "SYSTEM\CurrentControlSet\Control\Lsa"
strLsaValue = "Security Packages"
strSecProvKey = "SYSTEM\CurrentControlSet\Control\SecurityProviders"
strSecProvValue = "SecurityProviders"
strHostname = "."
modified = false
found = false

Set objWMI = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & strHostname & "\root\cimv2")
Set colItems = objWMI.ExecQuery("SELECT * FROM Win32_OperatingSystem")
For Each objItem In colItems
  strOSVer = objItem.Version
  SPlevel = objItem.ServicePackMajorVersion
Next
If Not Left(strOSVer,3) = "5.1" Then
  WScript.Echo "This script is only intended for Windows XP."
  WScript.Quit
End If
If Not SPlevel >= 3 Then
  WScript.Echo "Please install the latest Windows XP Service Pack from Windows Update."
  WScript.Quit
End If

Set objReg = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & strHostname & "\root\default:StdRegProv")
objReg.GetMultiStringValue HKEY_LOCAL_MACHINE, strLsaKey, strLsaValue, arrMultiRegSZ
size = Ubound(arrMultiRegSZ)
For i=0 to size
  If arrMultiRegSZ(i) = "tspkg" Then
    found = true
  End If
Next
If found Then
  WScript.Echo "tspkg already added to HKLM\SYSTEM\CurrentControlSet\Control\Lsa"
Else
  ReDim Preserve arrMultiRegSZ(size + 1)
  arrMultiRegSZ(size + 1) = "tspkg"
  objReg.SetMultiStringValue HKEY_LOCAL_MACHINE, strLsaKey, strLsaValue, arrMultiRegSZ
  modified = true
End If

objReg.GetStringValue HKEY_LOCAL_MACHINE, strSecProvKey, strSecProvValue, strValue

If Instr(strValue,"credssp.dll") Then
  WScript.Echo "credssp.dll already added to HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders"
Else
  strValue = strValue & ", credssp.dll"
  objReg.SetStringValue HKEY_LOCAL_MACHINE, strSecProvKey, strSecProvValue, strValue
  modified = true
End If
If modified Then
  WScript.Echo "Settings updated. You will need to restart for the changes to become active."
End If

Set objReg = nothing
Set objWMI = nothing