Ask A Cartographer

Batch Merge Shape Files

December 23 2010 | 4 comments
Categories: Map Data

I have a large quantity of UK shape files from the Ordnance Survey OpenOS Data catalogue (VectorMap District). The data is divided into 10km tiles and each tile has up to 14 different shape files (point, line and polygon). Each tile is in its own folder and all files are named the same way, ie. "Road_Line.shp" or "Height.shp" in each folder.
Is there a way that I can merge all "Height" or "Road" files in a batch, rather than having to manually select each individual file through the Merge tool. Is there a way of merging files with the same name, that sit within different sub-folders? I have looked at the batch option for Merge, but it does not appear to do what I need.

Mapping Center Answer:

Yes, you can merge in a batch mode using Python Scripting or Iterators in Model Builder. In python, you would need to loop through each subfolder and then merge the files. I have created a sample script for your case, which is attached here. 

You can modify the attached script in a text editor and then load it in a python window and run it. This script will merge all your roads/height shapefiles in one process.

Hope this helps!

different filenames posted by Dale Kunce on Jan 11 2011 3:31PM
Thanks for posting the response. If the files weren't named exactly the same how would you alter your python script? For instance my data is named "[county][year].shp" I would like to merge years into one shapefile for each county.
modify the script posted by Rajinder Nagi on Jan 13 2011 10:22AM
In this case, you can modify the script to check if a county name exists in a shapefile name and if it does exist, create a list of that and then do the merge. You would need to change the script logic starting at line 19 with something like this (the formatting is lost in the code below so you will need to add it back in):

# check if 'Riverside' name exists in shapefile name
if filename.find("Riverside") == 0:
print os.path.join(root, filename)
riverside = str(os.path.join(root, filename))
riversideList.append(riverside)
elif filename.find("San Bernadino") == 0:
print os.path.join(root, filename)
SanBernadino= str(os.path.join(root, filename))
SanBernadinoList.append(SanBernadino
elif
--- code here---

There are a number of ways to acheive what you want, and I am suggesting just one. You can read about arcpy here: http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/A_quick_tour_of_ArcPy/000v00000001000000/

and regarding python here: http://docs.python.org/
2013 Update posted by Dylan Kennard on Jul 12 2013 9:24AM
I am sure many people found this thread helpful (and I am a low level python user) so I wanted to add some details based on my experience in July 2013 with ArcDesktop 10.1:

The attached python script will not work in Python 3.x there is a huge library change over which does effect the os.path module. You will have to convert it or stick with the 2.7 version which is installed by default with 10.1. (So best you stick with 2.7)

My Goal:
I had about 1800+ shapefiles with 5 different themes clearly named in the file name in 1 folder and wanted to merge them based on theme into a FGDB before moving to SDE.

The Process I took:

Once I figured that my 3.x version of Python wouldn't work with this code I reverted to 2.7 and I updated the code to fit my needs and feedback from "Ask A Cartographer - Batch Merge Shapefile". Below is a subset of the code in Python 2.7

if __name__ == "__main__":
try:
SrcPath = r"C:\Data\TOBIN" ## Provide the path for data with subfolder here
for root, dirs, files in os.walk(SrcPath, topdown=True):
for filename in files:
if filename.find("TX") == 0: # check if 'XXXXX' name exists in shapefile name
print os.path.join(root, filename)
blk = str(os.path.join(root, filename))
absList.append(blk)

arcpy.Merge_management(blkList, txblocks)
print "script finished"

It got hung up and only grabbed the first file and stopped. Instead of troubleshooting for hours I jumped to the Iterator page mentioned in the original answer to see if model builder was an option. I am way better at that then Python. The iterators work great so I did the below somewhat manually.

Step 1) Used Windows Explorer wildcard search on the folder of interest for *abs. Cut all those resulting files and pasted into a stand alone folder called "abs". Did this 5 more times based on my themes
Step 2) Built a Model composed of "Iterate Feature Classes" --> "Collect Values" --> Merge Tool
Step 3) Ran the tool 1 at a time on each folder. Worked great and one layer took over an 1.5. Something beneficial from doing it in Model Builder vs Python (unless you code updates into your python) is that it tells you that it is still running. Seems like search for this because it is such large datasets, which could take hours.
Time Crunch posted by Dylan Kennard on Jul 12 2013 9:26AM
Oh yeah and Step 1 took me 30 mins.
Step 2 took me 30 mins
Step 3 - Sat back and relaxed while the computer did all the work.

If you would like to post a comment, please login.

Contact Us | Legal | Privacy |