Final Script to fix DataSources in ArcGis via ArcPy.

This is a followup to this previous post here, that describes how to implement a script that fixes datasources when moving an mxd from one work space to a new work space. Anyhow, after several days of running the initial script to fix ArcGis dataaources there were several enhancements or feature request submitted. The request are as follows, the script would need to move any data sources to a central location within the new work space. The script would also need to be updated to handle geodatabases.

While updating the script I took the initiative to make it more generic and I converted some constant variables to command line parameters so that it may be applicable to anyone. I also applied some clean coding standards for readability and good coding practice.

Anyhow the properties of the scripts or commandline parameters are as follows:

And I have taken the liberty to make the full working script available to anyone interested and can be found here.

Script has been updated, there was a couple of bugs with the copy method please read through the commit log for more information.

MITx 6.00 Review: Be more productive.

So this past February I had the pleasure of taking MITx's 6.00. Recently I noticed that edX is offering this course again this fall. So I'd like to share my thoughts and experience.

The knowledge and experience gained from this course is immeasurable. The course gives you a working knowledge of various algorithms used in programming to solve very complex problems. It also helps you learn python if you previously have never used the language or at least gives you a better understanding of it's mechanics and what makes it such a pleasurable language to work with. 

Before participating in the course I had only dabble in python and never really took the language all that seriously. Honestly, now I have a special reverence for the language. I believe it is the perfect tool for any engineer or scientist that wants to be more productive in their work. That is what makes the language and course so unique. It helps you be more productive, specifically if you are dealing with large amounts of data or iterative calculations, or dealing with some complex computations you might need to reuse from time to time.

The course is built for your success. It doesn't require you to be very knowledgeable about programming or about computers for that matter. The only requirement is really a thirst for success and knowledge, and perhaps an interest or need to solve some time consuming problems in a more efficient manner. 

So now that I have peaked your interest perhaps you have some questions that are rattling your brain. So feel free to add them to the comments and I will try and respond to them with the best of my abilities.

Oh I forgot to mention it might also be a good way to get MIT on your resume, since you will receive a certificate of completion if you receive a passing grade and abide by their Honor Code. Best of luck.

Updating DataSources in ArcGis via ArcPy when implementing a new filing layout.

Information:

To better understand the problem and solution, it is perhaps prudent you know a little about the old and new filing structure. Previously, all geological and GIS data where stored on one drive and engineering data on another, both drives used a specialized filing structure. The old filing system was cumbersome and difficult to understand. This made deleting or retrieving data for a particular job extremely difficult. The other problem was that any company owned or perpetual data was stored on either of each drive depending what the data related to.

So a new filing system has since been implemented. Now there is a knowledge drive and a project drive. The project drive holds all data related to a specific project whether it be engineering, GIS, or geological. The knowledge drive holds data that is owned by our company, and again the data could be administrative, GIS, geological or engineering.

The new filing structure makes it extremely easy to retrieve and look up data base on a project. It is also many times easier to purge all data related to a particular project. It also ensures that any data that is consider perpetual and needs to be on our system for long periods of time will not be accidentally removed when project data is purge.

The Problem:

For the most part our GIS personnel update existing ArcGis map files year after year, spoiling the previous year's work. Thus, I.T. is forced to retrieve data from tape for any scenarios involving data corruption or legal issues. This is contrary to policy that mandates any data for the past three years must be instantly retrievable. So policy now demands that our personnel copy old maps and GIS data to a new project location for returning clients. The issue with this particular scenario is that the links to the data sources don't get updated automatically when performing a copy and paste, and they are not relative. There is also the added difficulty that various data sources are being moved to different locations.

The Solution:

Please note all the code are examples and are more of a proof concept then an actual plug and play solution. The reason being that every company implements a different filing system, so the code has been adjust so that it may be applicable to a hypothetical scenario. 

First, lets fix any data sources that might be relative and were moved when the current map data was copied from the existing location to the new location.


mxd = arcpy.mapping.MapDocument("CURRENT")





OldWrkSpace = arcpy.GetParameterAsText(0)
NewWrkSpace = arcpy.GetParameterAsText(1)

mxd.findAndReplaceWorkspacePaths(OldWrkSpace, NewWrkSpace)

Next, lets fix any data that might be in found in the knowledge drive.

filesInNewDrives = [] 
oldDriveLetter = ""
knowledgeDriveLetter = ""

# Within a string search for char, return array of it's indexes. 
def FindChar(s, ch):
    return [i for i, ltr in enumerate(s) if ltr == ch]

# Builds a list of all files within a particular directory, this is done recursively.
def GetAllFilesInDir(dirToSearch, dirSearchResults):
    for root, subFolders, files in os.walk(dirToSearch):
        for file in files:
            dirSearchResults.append(root + "\\" + file)


# Search for a file name, in a list of file paths.
def SearchForFileInList(dataSourceFile, listOfFiles):
    for afile in listOfFiles:         
        if (dataSourceFile in afile):
            return afile
    return ""

# Return the data source file, and parent directory.
def GetDataSourceFileName(currentDataSource):
    indexOfSlashes = FindChar(currentDataSource, "\\")
    return currentDataSource[(indexOfSlashes[-2]+1):]

# Return the data source file name only.
def GetDataSourceShortName(currentDataSource):
    indexOfSlashes = FindChar(currentDataSource, "\\")
    return currentDataSource[(indexOfSlashes[-1]+1):]

def GetDataSourceWorkSpace(dataSource):
    indexOfSlashes = FindChar(dataSource, "\\")
    return dataSource[0:indexOfSlashes[-1]]

mxd = arcpy.mapping.MapDocument("CURRENT")

GetAllFilesInDir(knowledgeDir, filesInNewDrives)

for lyr in arcpy.mapping.ListLayers(mxd):
    if lyr.dataSource.startswith(oldDriveLetter):
       dataSourceFileName = GetDataSourceFileName(lyr.dataSource)
       arcpy.AddMessage("Searching for data source file name: " + dataSourceFileName)
       dataSourceNewFileLocation = ""
       dataSourceNewFileLocation = SearchForFileInList(dataSourceFileName, filesInNewDrives)
       if (dataSourceNewFileLocation is ""):
           dataSourceFileName = GetDataSourceShortName(lyr.dataSource)
           dataSourceNewFileLocation = SearchForFileInList(dataSourceFileName, filesInNewDrives)
       if (dataSourceNewFileLocation != ""):
           arcpy.AddMessage("Found data source file at: " + dataSourceNewFileLocation)
           lyr.findAndReplaceWorkspacePath(GetDataSourceWorkSpace(lyr.dataSource), GetDataSourceWorkSpace(dataSourceNewFileLocation))


Finally, move any data source not found in the new project directory or the knowledge drive to the new project directory.

def copy(src, dest):
    try:
        shutil.copytree(src, dest, ignore=shutil.ignore_patterns('*.lock', 'UTM_ZONES.shp'))
    except OSError as e:
        # If the error was caused because the source wasn't a directory
        if e.errno == errno.ENOTDIR:
            shutil.copy(src, dst)
        else:
            print('Directory not copied. Error: %s' % e)

mxd = arcpy.mapping.MapDocument("CURRENT")

destinationDir = NewWrkSpace + "\\DS"

for lyr in arcpy.mapping.ListLayers(mxd):
    indexOfSlashes = find(lyr.dataSource, "\\")
    dataSourceName = lyr.dataSource[0:indexOfSlashes[-1]]
    arcpy.AddMessage("Fixing by copying datasource " + GetDataSourceWorkSpace(lyr.dataSource))
    # Ensure data source doesn't already exist on project directory
    if not (os.path.exists(destinationDir + dataSourceName)):
        arcpy.AddMessage("Copying " + GetDataSourceWorkSpace(lyr.dataSource) + " to " + destinationDir + dataSourceName)
        copy(GetDataSourceWorkSpace(lyr.dataSource), destinationDir + dataSourceName)
        lyr.findAndReplaceWorkspacePath(GetDataSourceWorkSpace(lyr.dataSource), destinationDir + dataSourceName, False)
        
    
Please remember this is only a skeleton of the final solution, the code is not meant to be used straight out of the box, there are modification that are required and the final solution included more logic to prevent some errors from occurring.

Fix for Microsoft and Google's Battle over Sync

Users' of Windows 8 and Google services by now have heard of the great fiasco that these two giants have caused for their loyal users. For those people who have been living in the dark ages, and have yet to hear about all the commotion I will break it down for you.

The apps affected by the battle raging between these two are basically Mail, Calendar and People (contacts). As of January 31, 2013 Google shut off it's Microsoft Exchange ActiveSync (EAS) Service for new users. Existing users were supposed to be able to continue to use the service in order to sync the three applications with their respective Google services without any trouble. In late March of 2013, though Microsoft released an update that broke all EAS support for Google customers, even the ones that were expected to have continued service as promised by Google.

Now, there are various workarounds documented on the net  to get Google services working with Mail, and People again, but none of worth for Calendar. Here I will outline an old trusted service that I used in my Blackberry days to get Google Calendar to two-way-sync with Windows 8 Calendar. This service could also be used with Mail, and People. And now the moment you have all been waiting for... The service is called Memotoo. I have to be honest it had been awhile since I had logged in to my account, but was I ever surprised and grateful the company was still around and that the service had flourished and improved.

Anyhow here it is, How to sync Windows 8 and Google Calendar:

Head over to Memotoo and Sign Up for an account

Add your Google account as one of your accounts to sync under Calendar and Contacts.

Once signed in, head over to calendar, and on the top left corner click the sync icon.

On the left hand side click "Windows 8".

Finally just follow the instructions to completion, and that should have you syncing your google account with your Windows 8 pc or tablet.

Get CPU Temp To Display In XBMC.

Install the "lm-sensors" package:
# sudo apt-get install -y lm-sensors

Identify the installed sensors and load the required modules:
# sudo apt-get install -y lm-sensors
You can press enter for each of the questions, except for the last one which is "Do you want to add these lines automatically?" where you should type "yes" and hit enter.

Reboot your device.

Verify sensors is working:
sensors -u

Edit your advancedsettings.xml and insert the following line:
echo "$(sensors -u | grep "temp1_input"| awk '{print $2 }' |awk '{printf("%d C \n",$1 + 0.5);}')"
XBMC should now detect the temperature of your cpu cores.

Update: WOL for ZOTAC HD-ND22 and Ethernet Fix for XBMCLive

/etc/network/interfaces now looks as follows:


You will notice there has been a couple of modifications.

1. pre-up /usr/sbin/ethtool -s eth0 speed 1000 duplex full
   

   This line helps resolve issues where the Ethernet connection is not always active on Boot.

2. pre-up /usr/sbin/ethtool -s eth0 wol g
   

   This line is used in lieu of ethernet-wol g
   for the sake of connection stability on boot.

/etc/default/grub was also edited to remove acpi_enforce_resource=lax since it no longer seems to aid in any beneficial way.


And finally last but not least /etc/rc.local was modified to look as follows:

Compiling latest svn of XBMC for Nvidia Hardware.

The svn ppa does not get updated too often so sometimes it is nice to be able to build your own svn from source. I will try and explain the simplest method of achieving this.

First add the svn ppa:
sudo add-apt-repository ppa:team-xbmc-svn/ppa

Next fetch the essential build tools and all of XBMC's required libraries and headers:
sudo apt-get update
sudo apt-get install build-essential
sudo apt-get build-dep xbmc

The next step is to get the latest source code for the most recent svn. To do this subversion must be installed, and can be done so by running:
sudo apt-get install subversion

Then checkout the latest svn source code by executing:
cd $HOME
svn co https://xbmc.svn.sourceforge.net/svnroot/xbmc/trunk/ xbmc
The above command will place a copy of the source code in a directory named "xbmc" in the home directory.

Within the xbmc directory that the source code is contained, run the following commands in order to compile it:
./bootstrap
./configure --enable-vdpau --disable-pulse --disable-crystalhd
make -j2
make -C lib/addons/script.module.pil
make -C lib/addons/script.module.pysqlite
sudo make install
Replace the number 2 with the number of cores in your processor.

Now make a backup of the original XBMC install, and make the recently compiled version the default one to boot.
sudo mv /usr/bin/xbmc /usr/bin/xbmc.bak
sudo mv /usr/bin/xbmc-standalone /usr/bin/xbmc-standalone.bak
sudo ln -s /usr/local/bin/xbmc /usr/bin/xbmc
sudo ln -s /usr/local/bin/xbmc-standalone /usr/bin/xbmc-standalone

Almost done, reboot:
sudo reboot

Reinstall nVidia Drivers:
sudo apt-get install --reinstall nvidia-current nvidia-current-modaliases nvidia-settings