Organizing CAD Data In ArcGIS: Part 7/7 Automated Sorting

Automated Layer Sorting

I just got back from a trip to Albany/Schenectady New York. I would like to nominate The Pizza King of New York in Schenectady, NY as the best pizza I have ever had. The appetite I had from missing breakfast and the cold air, combined with some of the best pizza anywhere, resulting in a truly memorable culinary experience. When you walk into this whole-in-the-wall restaurant with its non-descript glass front you are suddenly struck with the magnificence of this functioning pizza museum. Every square inch of the preparation area, serving area and counter is covered with pizzas; a spectacular array of diverse pizza creations. There must have been fifty pizzas on displayed and ready to serve. Behind the counter was the Pizza King querying the patrons for their selections. Your chosen slices where cut from the pie and immediately transferred to the warming oven. By the time you were finished with ordering drinks and finishing the purchasing transaction, your warmed pizza was presented to you to enjoy. Like a well oiled machine the customers where served a great product with a smile, and in a timely fashion; true pizza precision!

Like different types of pizza at the Pizza King, the layers of a CAD file can be organized in ArcGIS as separate map layers from which you can take slices using various ArcGIS query and analysis tools. And, just like the Pizza King it is a good idea to automate your workflow and provide yourself easy access to those slices. In part 5 of this series I talked about breaking out different CAD layers as separate ArcGIS feature layers using a definition query. In part 6 I talked about saving these definitions into a separate ArcMap layer file. I have built two sample tools that can be found in a CAD Sample Toolbox that I created that can help you automate the process of sorting CAD files and defining layers or feature classes based on distinguishing CAD properties, such as CAD LAYER or COLOR or any supported CAD attribute.

The first tool is called SplitByColumn. The tool has general non-CAD-specific functionality to take any input feature layer including CAD feature layers and create new feature classes based on unique instances of values in a particular column. This is particularly helpful when you want to create a separate feature class for each CAD layer, or a separate feature class for each unique CAD line style, etc…, but it is equally useful if you want to separate a feature class into multiple feature classes based on common column values.

The other tool CreateLayerFilesByColumn is essentially the same tool, but instead of creating an output feature class it create ArcMap Layer files which especially in the context of CAD files can be a useful way to generate input to ArcMap that organizes the CAD data without converting it to an ArcGIS feature class.

Organizing CAD Data In ArcGIS: Part 6/7 Layer Files

ArcGIS Layer Files with CAD

My daughter had 15 guests at her 13th birthday/"sleep"-over party this weekend, it goes without say that there was very little sleep happening. When the last parent came to pick up her daughter, she pulled up in her SUV and left the car running, while making a quick run to our front door to ring the doorbell (much to the dismay of her two small poodles). Like popcorn the dogs eagerly beckoned the return of their loving owner… Guess what happened…? Classic! The two dogs had managed to hit the automatic door locks; keys in the ignition, windows up, purse, cell phone inside and the car running! Back into the house she came to call AAA. I couldn’t help but share my appreciation of the humor in the situation, with her. The feeling was in no way reciprocated.

Then unexpectedly her daughter screamed and ran out the door, back to the car! I followed, not knowing why she had screamed; thinking to myself that the dogs had somehow managed to put the car in gear! As fate would have it, the dogs, now feeling the elevated tension of the situation had increased their activity inside the car and had managed to activate the automatic windows opening them about four inches! The little girl’s arm was not long enough to take advantage of the opening. And, she was afraid the dogs (now insanely active) might close the window on her arm. Luckily I was able to reach into the open window and unlock the doors before that eventuality!

Even with an electronic key and sometimes especially with automatic key/locks it’s worth considering a hide-a-key. In my last post I talked about how a definition query can be used to organize/filter the information in a CAD file and make it more useful in ArcMap. That information is locked inside the ArcMap Map document (.mxd) as part of the layer definition of the Map’s table of contents (TOC). What if you want access to that information outside of the map documents? (with or without poodles!)

To make the definition query information portable and accessible outside of ArcMap, say ... as input to a geoprocessing command, you might consider saving the ArcMap CAD layer with its definition query to a ArcGIS Feature layer file (.lyr). To save any ArcMap layer with its definition query and any table joins to a layer file (.lyr), right-click on the feature layer in the TOC and select the Save As Layer File… option.

You can also create an ArcGIS feature layer file (.lyr) with CAD data using the SAVE TO LAYER FILE geoprocessing tool from an existing ArcMap Layer or by using the combination of the MAKE FEATURE LAYER and SAVE TO LAYER FILE geoprocessing tools using the CAD feature classes as input.

Continue to Part 7

Organizing CAD Data In ArcGIS: Part 5/7 Definition Query

The Feature Layer Definition Query

In the last post I changed the default symbology in the CAD feature layer. In this way I could distinguish between different CAD features based upon the way they look. However, if I want to independently control, for example, the visibility all the water lines in a CAD drawing separate from the right-of-way lines that are also displayed in the same drawing, I need a separate ArcMap feature layer to represent just the water lines in the Table of Contents. Or, I may want to use ArcGIS spatial analysis tools to check setback distances from the right-of-way lines.

I wrote in a previous post that a CAD drawing is treated as a spatial database in ArcGIS. To make sense of the raw data in this type of database it is useful to create custom views that organize data in useful ways. These database VIEWS may include the joining of tables, and filtering of data to see a useful subset of the entire database. In ArcGIS a spatial database VIEW is most analogous to an ArcMap Feature Layer. When you add a CAD feature class or any feature class to ArcMap an ArcMap feature layer is created in the ArcMap TOC (Table of Contents). The data source for the feature layer is the feature class. You can have multiple feature layer entries in the TOC that all point to the same feature classes.

ArcMap feature layers can have their own display name. Feature layers including CAD feature layers, support table joins and a definition query. The definition query allows you to filter the source feature class and include just those features that satisfy the definition query. I add the CAD POLYLINE feature class multiple times and then rename the feature layer to represent the different systems within a single CAD file. I supply different definition queries to isolate each different data system in the CAD file for example:

LAYER = “WATER”

Using the CAD POLYLINE feature class as the source data to multiple feature layers I include definition queries to assemble a collection of independent data systems rather than a single collection of all the linear features in the drawing.

Use the ADD DATA tool to add the same CAD feature class multiple times to your ArcMap map. You can rename an ArcMap feature layer on the General Tab of the Properties dialog box. Renaming the feature layer give you a chance to describe the contents of your filtered data; a name like Water Lines. The definition query is entered on the Definition Query Tab of the same Properties dialog box. The the definition queries created in this way are only stored within the current ArcMap document (.MXD).

Continue to Part 6

Organizing CAD Data In ArcGIS: Part 4/7 Unique Values

Unique Symbols by CAD Attributes

Seeing everything all at once can be confusing, especially if everything looks alike. The saying "I can't see the forest through the trees!" can be very relevant when using CAD files as GIS content.

If you consider an AutoCAD or Microstation file as a spatial database of features, then the ArcGIS CAD Feature Dataset could be considered a database VIEW of that spatial database. Like a tabular database that contains many tables and records of raw data, an AutoCAD or Microstation contains raw spatial data.

When you add a CAD feature class to ArcMap by using the Add Data Tool, you can browse the file system to find the blue folder CAD Feature Dataset representation of a CAD file that in turn contains the five standard ArcGIS CAD feature classes, POINT, POLYGON, POLYLINE, ANNOTATION and MULITPATCH. Because ArcGIS tools rely on a homogeneous data type, (all points or all lines or all polygons, etc…) the simplest level of organization is to sort the entities into feature classes based on this fundamental characteristic; geometry type.

The default symbology chosen by ArcGIS to display the Microstation or AutoCAD CAD feature classes added to ArcMap is like any other GIS feature class. ArcGIS will symbolize all the features in the feature layer with a single symbol selected at random. This monochromatic single symbol representation doesn’t provide any clues to the potential meaning of the CAD objects. Like any GIS feature layer you will want to symbolize the CAD feature class in a meaningful way to convey some cartographic message. With CAD feature layers, the CAD properties that the CAD file was drawn with originally provide an excellent set of attributes upon which you can symbolize the map.

I generally find that using the CAD COLOR property or the CAD LAYER property as the column upon which to perform a unique symbol rendering of the CAD file gives you the most distinction between all the different types of lines or polygons in a CAD feature class. If you have used a lot of AutoCAD BLOCKS or CELLS in your CAD file, you may want to consider using the block name stored in the TEXT field of the CAD feature attribute table, as the column value upon which you perform a unique value rendering of the layer.

To change the symbology of a CAD feature layer or any feature layer in ArcMap, I right click on the layer in the ArcMap table of contents and then select the properties tab. Select the Symbology Tab, from the under the Categories heading in the Show: pane, select the Unique Values option and then make your choice of COLOR, LAYER, TEXT etc… from the Values Fields pull down list.

The result of changing the symbology of the CAD feature layer will display the CAD features with different symbols based on the different CAD layers, or CAD colors or whatever distinguishing column you have selected.

Continue to Part 5

Organizing CAD Data In ArcGIS: Part 3/7 Entities as Features

The Reading Entities as Features

My father is a retired forester. He told me of a comment a surveyor friend once made. This surveyor told my father there are only two kinds of trees…stick trees and Christmas trees! As far as this surveyor was concerned that was all that mattered when identifying a tree. The only important distinguishing feature worth noting was whether it looked like a Christmas tree, or not. When ArcGIS sorts CAD entities in a drawing a similar albeit more useful categorization occurs.

ArcGIS doesn’t attempt to present CAD data using its free form CAD data structures, instead it overlays a standard GIS spatial database schema on the CAD file. CAD entities are sorted into different GIS feature classes and are thus recast as GIS features. By recasting the CAD entities as GIS features CAD data becomes a true GIS data source. All of the tools in ArcGIS designed to work with GIS data, work with CAD data because the first thing ArcGIS does when reading a CAD file is to organize the data into an ArcGIS CAD feature dataset of 5 standard feature classes (POINTS, POLYLINES, POLYGONS, ANNOTATION and MULIPATCH).

The resulting CAD POINT feature class will be built from all the CAD entities
and in the CAD file that have a single coordinate as their primary defining geometry. This would include AutoCAD BLOCK INSERTS, POINTS and SHAPE entities or Microstation CELLS, POINTS, etc… In the case of BLOCK INSERTS/CELLS there may be sub entities that are also sorted into other feature classes.

The CAD POLYLINE feature class will contain all linear entities and sub entities along with their true curves geometry; ARCS, LINES, POLYLINES, 3DPOLYINES, LWPOLYLINES, LINE STRINGS, etc… This feature class will also contain closed shapes such as TRACES, CIRCLES and closed POLYLINES.

POLYGONS are built from all closed shapes. These closed shapes will also be represented in the POLYLINE feature class so that you have the ability to treat closed shapes as loops or as areas. These CAD entitles would include CAD polygons, closed POLYLINES, TRACES, ELLIPSES, CIRCLES, etc…

The CAD MULTIPATCH feature class is comprised of CAD 3DFACES. These special ArcGIS polygon features are visible in 3D using ArcGlobe or ArcScene.

All CAD TEXT entities are included in the CAD ANNOTATION feature class. Their insertion point is uses as their primary geometry. The ArcGIS CAD ANNOTATION feature class can be used as POINT input to may ArcGIS tools.

All CAD feature classes have a standardized table of attributes. This virtual read/only attribute table is assembled from the CAD graphic properties of the CAD entities (COLOR, LINESTYLE, THICHNESS, LINEWEIGHT, etc...). This table also includes any Microstation TAG values and AutoCAD BLOCK ATTRIBUTE values. Using information in the table you can query and further sort the data.

Continue to Part 4

Organizing CAD Data In ArcGIS: Part 2/7 Map Layer

CAD Files as a GIS Map Layer

My two year old has two distinct uses for her big bucket of blocks. One use is to dump out the contents of the blocks onto the floor and then meticulously stack them into a monolithic tower. The other use is to take the bucket of blocks as a whole dump it upside down and use it as a stepping stool to wreak havoc on my wife’s desk, and to retrieve snacks from the counter top!

ArcGIS has two distinct ways to represent CAD drawing data. One way is to treat the entire CAD file as a single cartographic layer. The other way is to treat the CAD file as a container of geographic features.

A CAD map product is often useful in its entirety as an information layer. The CAD document tells a story all by itself and can be useful as-is for its cartographic content. ArcGIS has the ability to represent an entire CAD file as a single cartographic layer where all of the vector objects and text are included on a single layer. The CAD graphic properties are used to draw the CAD file in ArcMap. In this way an ArcGIS CAD Drawing Layer is not unlike a raster image in that it is the picture of the entire CAD file that acts as a valuable data layer. The CAD Drawing layer mimics the CAD symbology and provides a single entry in the ArcMap table of contents for easy visibility control. The CAD object representation is still as vectors and objects can be selected and identified. However, spatial analysis and other advanced query operations are only possible with the GIS feature representation of a CAD file provided by the ArcGIS CAD Feature Dataset representation of a CAD file. I will discuss the ArcGIS CAD Feature Dataset representation in a future post.

We can simply navigate to a CAD file and select it to be added to a current ArcMap map. In previous posts I have discussed the affects and usefulness of defining a spatial reference. If a companion ArcGIS projection file is found in the directory with the selected CAD file a CAD file will be displayed on the map according to its location on the globe relative to other data already in the map frame. If no projection file is included the CAD file will be positioned according to the coordinates unmodified in the CAD file relative to whatever the coordinates are currently in the map frame. Since ArcGIS has no other information it assumes the CAD files coordinates should be displayed according to the map current frame’s coordinate system.

In a previous post I also discussed the use of an ESRI CAD world file, used to modify the coordinates in a CAD file to perform any need transformation of the coordinates that would relocate, scale and rotate data drawn in a relative coordinate system for use in a geographic coordinate space.

When browsing for CAD files you will notice two entries in the Add Data dialog or in the ArcCatalog data browsing pane. The white compass icon denotes the CAD Drawing Layer representation of an entire CAD file and the blue folder icon is used to access the CAD Feature Dataset, which include the five standard ArcGIS CAD feature classes.

Once the white drawing icon is selected all of the objects in the CAD file are displayed as a single layer in the table of contents. Using the layer properties dialog box for the CAD Drawing layer you can modify the transparency and some other display characteristics such as renaming the Map layer and controlling the visibility of individual CAD layers in the CAD file. The symbology however is fixed to that found in the CAD file. Otherwise the CAD Drawing Layer acts like any other ArcMap layer. You can move the layer up and down in the drawing order by dragging it and toggle its visibility for the entire layer.

Using the CAD Drawing Layer is most useful when the CAD file is itself a complete cartographic representation of the data useful as-is for plotting or browsing the data.

Continue to Part 3

Organizing CAD Data In ArcGIS: Part 1/7 Tools for CAD Data

Smart GIS tools for Dumb CAD Drawings

I thought I’d try a provocative title for sweeps week. Or, how about another title… “Making Sense of this CAD Spaghetti, That Your Meatball Intern Has Drawn!" …Now I’m getting hungry.

The truth be told CAD data can be as smart or dumb as you want it to be. This sometime contributes to the problem, some CAD drawings are very organized and adhere to strict standards and logical data constructs. Other CAD drawings can be either ambiguous, or could be called “creative” at best. The science of reading CAD drawings in a GIS application is founded on the premise that CAD files are a collection of GIS features with various CAD properties that can be used to distinguish one from another. However it would be presumptuous for the GIS to ascribe meaning to the CAD objects (see series on Semantic Translation). Organizing CAD data by layers is good thing but so is using layers to define subtypes or entire data systems that might include points, line and area features.

One of the foundations of GIS is the fact that it stores simple standardized data structures and uses powerful spatial query and analysis tools to perform smart tasks. Therefore, it makes the most sense for GIS to view a CAD drawing as simple set of features with attributes and then use the powerful GIS toolbox on those features as if they were native GIS features. Even though a single CAD drawing can have the complexity of an ArcGIS Map document with hundreds of included features classes, ArcGIS sees a CAD drawing as a data set of only five feature classes. Initially the only criteria for differentiation is geometric type (POINT, POLYLINE, POLYGON, ANNOTATION and MULTIPATCH). All of the linear features of a CAD file can be accessed in ArcGIS through that CAD file’s CAD POLYLINE Featureclass. All CAD Points, Blocks/Cells and other point type CAD Entities in a CAD file can be accessed in ArcGIS using the CAD POINT Featureclass.

How you choose to organize the contents of the CAD Feature classes is up to you or rather up to the data constructs, standards and workflow task you are attempting to solve. For example, you may want to deal with only the Right of Way lines in a CAD file as a GIS dataset or maybe Parcel Block lines. However, you may also want to consider using the right of way lines, block lines and lot lines to build polygons to depict parcels from the CAD data. Choosing how you view and organize your CAD drawings may initially require more attention, but having the flexibility to apply a customizable organizing filter to your drawings provide additional problem solving capability.

The concept of abstraction is key to geographic information systems, feature type standardization is another. Using a points, lines and areas to depict geographic features allows the same tools that work on sewer systems, to work equally well on bicycle trails... cities and wells, airports and elephant breeding grounds.

This series will be dedicated to techniques to organize CAD data in ArcGIS to make it the most usable for various workflow tasks.

Continue to Part 2

AutoCAD Spatial Adjustment Sample Tool

Whether you consider yourself a Steelers fan or a Seahawks fan I hope you enjoyed the “big game” this past weekend. I was intrigued by a Super Bowl conversation I heard here in the hallway from a colleague. Turns out this self proclaimed diehard Steelers fan, found himself rooting for the Seahawks on a number of occasions to help him hit his numbers for the office pool!

You’ve heard it said, “…when all you’ve got is a hammer every problem looks like a nail!” There is also much talk about using the right tool for the job, best-of-breed tools, etc... Often times our workflow spans what might be considered two different software domains, where the right tool for the job is BOTH!
GO, ALL STARS!

One form of GIS and CAD interoperability often overlooked is the API’s common to both GIS and CAD. VB Automation provides a compelling environment for accessing the tools and data structures of both GIS and CAD in a single application. The Savvy automation programmer can create his own Hybrid GIS CAD application choosing from the toolboxes of both GIS and CAD to perform a given task. In some ways it is the purest form of GIS and CAD interoperability. It is in a sense running GIS and CAD at the same time on the same data.

Last post I eluded to the creation of an application that would assist in inclusion of an ESRI projection file for your CAD drawings and also eliminate the need for a ESRI CAD World file by using AutoCAD itself to perform the required coordinate transformation using the ESRI World File coordinate transformation definition. ...Well, happy belated Groundhog's Day, here is an application that does just that. AutoCAD World File Adjustment And Projection Files

This application built with VB automation tools will take as input any number of CAD files that either need to be adjusted using a World file, or that need a companion ESRI projection file. I mentioned in my previous posts that the World File is only needed if you don’t have or know how to use AutoCAD or if you are not permitted to “fix” your CAD data to put it into the proper coordinates. I also mentioned that an ESRI projection file doesn’t modify coordinates but rather identifies what coordinate system on our globe the coordinates in the file are recorded in. A projection file is optional but strongly recommended when working with CAD data in ArcGIS. The world file is useful in modifying the coordinates of CAD data without changing the actual data in the file. However with this tool we WILL change the data in the CAD files so that a World File would no longer be required.

One reason for automating this process in this software sample tool is that you may have hundreds of drawings that you need to modify or include projection files for.

This application takes as input multiple CAD files. It is assumed that all of the CAD files you are selecting need to be processed in some way. This tool will use any existing ESRI world files, but will use the default coordinates to perform an AutoCAD MOVE, SCALE and ROTATE operation on all other files. In this way you can either modify many CAD files with their existing world file transformations or use many CAD files that require the same transformation all at one time. You have the ability to save the AutoCAD files to the same name as the input thus overwriting the existing file with the modified data, by entering a null string for the Output CAD file name prefix. You do this at your own risk. This tool is only provided as a sample and may not work for you or your workflow. If you do overwrite your data the world file will be renamed to hide it from ArcGIS so that the transformation is no-longer used when working with the CAD files.

Another use of this tool is to copy ESRI projection files for selected CAD files. If projection files already exist it is assumed they are correct and will leave them. If however a CAD file does not have a companion ESRI projection file this sample tool is design to copy the selected ESRI projection file and name it appropriately to be recognized by ArcGIS.

The tool runs with AutoCAD in the foreground to help you monitor its progress or in the background.

The Complete CAD Spatial Reference

I have done my share of painting and have found that people take one of four approaches to painting around outlets. 1. Paint over everything, I guess this usually starts by forgetting that the electrical outlet is there, then trying to use the roller to go around it accidentally painting over it and then saying to yourself, that doesn't look that bad doesn’t it? (it does look that bad!) 2. Use a brush to cut around the plates; this usually results in a ridge around the outside of the plate and a small sliver of the underlying color showing through. 3. Tape around the switch plate and then paint. This is the worst one, although depending on your skills of taping (and more importantly removing the tape) it can create a nice finished result; however, it is ten times more work than is needed. 4. The best practice for painting around switch plates is to remove the plates, paint and then after everything is dry put the plates back on. (Same can be said for light fixtures, air vents, etc...)

Best practices for CAD interoperability can take many forms since the use of CAD and GIS data can be very diverse. Last post I discussed the fact that it is always better (although not required) to include a coordinate projection file with your CAD drawings to define where on the world you are. If you don’t you’ll get a little annoying message telling you that the data you’ve added to ArcMap has an unknown spatial reference. Besides that ArcGIS won’t know what to do with your CAD data if it is added or copied to a map frame with a different coordinate projection. ArcGIS will use the coordinates in the drawing unmodified unless you give it more information about where on the globe the data belongs.

Another file that can be included with a CAD drawing is called an ESRI world file. This file is an ASCII file of two sets of coordinates that define a coordinate transformation. The file containts two base points and two target points. If a world file exists in the same directory with the same name as the drawing with a .WLD file extension. The four points in the world file will be used to calculate new coordinates for the drawing. With these four points there is enough information to define a MOVE, SCALE and ROTATE transformation. When data is created in page units or some local coordinate system the world file can be used to push the CAD data into its proper position. If you have 500 drawings that are all oriented correctly one to another, but are drawn in a local coordinate system, say using an origin coordinate (0,0) at the corner of some building, you can create an ESRI world file that defines the proper geographic location of that building corner (0,0) and one other point on the map and then include what the proper coordinates should be. Since each of the drawings is properly oriented one to another, you can save time by simply copying the one world file 500 time renaming it for each file name, with a .WLD extension. By also including a projection file properly named for each of the 500 CAD files, ArcGIS would have a complete and accurate spatial reference for the data in each CAD file even though the data was drawn in a local coordinate system. Better than creating a world file for each of these CAD drawings would be to use your CAD program to MOVE, SCALE and ROTATE the drawing contents into the proper coordinates space so that the ESRI world file would not be needed. World files are for people who don't have or know how to use their CAD program.

What if there was a sample tool that could control your AutoCAD application and use a single ESRI world file definition to update all your CAD drawings for you? What if this tool also allowed you to pick a spatial reference (projection file) for all of them and create the properly named file? Would that be useful to you…? Send me your comments.