Sorry for the long post, but I thought this might be helpful to add. I originally prepared this for the National Broadband wiki group last spring when we were doing our last broadband update. Haven't checked in detail on an update, but I think most if not all is still up to date....
Broadband Map metadata in ArcGIS 10 – Lessons learned 3/25/2011
There is no better topic on a Friday afternoon than metadata, right? For those grantees that are struggling with the metadata for the spring 2011 submission in ArcGIS 10 like we were, I thought I would share what we painfully learned. I’m not an authority on metadata, and FGDC metadata in version 10 is still a bit problematic, so don’t take these tips as authoritative, but I hope they help.
We have converted to ArcGIS 10, service pack 1 and don’t currently have a workstation with version 9.3. All our processing for the spring submission was done in ArcGIS 10, so our choices were to uninstall 10 on one of our workstations and load 9.3 in order to do metadata, or take the plunge into version 10 metadata. We chose to take the plunge, naively thinking the water would be at least lukewarm. Little did we know what we were getting into…
The NTIA and FCC team reversed their decision to require an FGDC style standalone XML metadata file between the January and March, 2011 webinars (they had experienced some of the issues we discuss here) and decided in the end for this submission to require just five basic fields (you can find those in the checklist published March 24, 2011). So technically, if you are using version 10, the advice was “you are on your own, do your best”.
If you are still using ArcGIS 9.3, you don’t need to read any farther, just do your metadata like you did for the last submission. If you are using version 10 and don’t want to read all the details, I’ll cut to the chase here. Turns out you can edit your metadata in Version 10 using the same visual basic FGDC metadata editor as in 9.3 as long as you have service pack 1 installed, read a few blogs, and install an add-in and a patch (instructions below). You can also create/edit metadata in the new native ArcGIS Metadata Editor and export an almost compliant FGDC style metadata XML file (instructions below). We chose the latter, since we want to add cumulative process steps for our metadata for each submission. We also like to have our metadata reside with the feature class, not in a separate file. We also do not want to reinvent or recreate the metadata twice a year. Once you get it set up in the new ArcGIS metadata format it is a lot easier to maintain and edit, and theoretically export in multiple standard formats.
On a related side note, the March 24 checklist includes a best practice to not import your previous metadata into the new data model. I disagree with this. There is no problem importing your previous metadata into the new data model or a new feature class built with the data model, as long as you save a copy of the empty data model and copy the relevant items that NTIA/FCC entered in the metadata and want you to include. Unless you have really sparse metadata, or want to start from scratch each time, it is less work to import your previous metadata and then cut and paste the NTIA required text into it. In fact, it didn’t appear to us that anything was changed in the descriptions, use constraints, etc. between the October, 2010 and the January, 2011 data model metadata from NTIA, except for the attributes and entities that are auto populated from the feature class itself.
I did a little searching to see what was really required for the broadband mapping program, and found the following in footnote 47 talking about the technical appendix in the July 8, 2009 Federal register notice.
(NTIA should create Metadata (data about the data) according to Federal Geospatial Data Content (FGDC) standards to be generated after geo-coding); State of Arizona Government Information Technology Agency at 9 (NTIA should create Metadata (data about the data) according to ESRI mapping standards); CostQuest/ Link America Alliance at 18, 19 (maps and features (data layers) should be collected in accordance with Open Geospatial Consortium (OGC) standards for geospatial data).
I didn’t dig any further, there are likely other references. We have assumed that an FGDC standard would be required for our metadata submission. The federal partners and perhaps those grantees more knowledgeable than I on metadata standards can provide more thorough information, but as I understand it the FGDC CSDGM standard is more of a defacto standard than a formally approved one. The North American profile of ISO 19115 2003 standard has been adopted by several (maybe all?) federal agencies. Regardless, we assumed we needed to provide the metadata in a manner that could at least be exported to the FGDC CSDGM format. Here are the background and technical details.
There are good explanations for the conceptual background on why ESRI changed things up in the metadata processing in the ArcGIS 10 software release. It boils down to the ongoing phasing out of visual basic (due to Microsoft phase out) combined with the environment of evolving standards and a better way to deal with many standards. In essence, ESRI changed the paradigm and re-wrote the interface and metadata editor, then added some crutches to “help” ease the transition. There are some URL’s for blogs sent by ESRI technical support that describe the transition in three installments. The fourth URL tells most, but not all, of the process to actually implement the transition for FGDC metadata from 9.3 to 10. I followed these URL’s with some of my questions to ESRI support, and the answers follow that.
There are still some issues I haven’t figured out. There are also some known bugs that prevent a few of the required FGDC fields to be exported that will supposedly be addressed in service pack 2 (not out yet). ESRI is still looking for some additional reference documents that might help with the transition (I’ll post those when I receive them). That said, it is at lease sufficient for our needs, and goes well beyond what is being required for the Spring, 2011 submission.
http://blogs.esri.com/Dev/blogs/arcgisdesktop/archive/2011/01/03/A-new-approach-for-Metadata-with-ArcGIS-10.aspx
http://blogs.esri.com/Dev/blogs/arcgisdesktop/archive/2011/01/06/A-new-approach-for-Metadata-with-ArcGIS-10-_2800_part-2_2900_.aspx
http://blogs.esri.com/Dev/blogs/arcgisdesktop/archive/2011/01/11/A-new-approach-for-Metadata-with-ArcGIS-10-_2800_part-3_2900_.aspx
http://blogs.esri.com/Dev/blogs/arcgisdesktop/archive/2011/02/24/Configuring-ArcGIS-for-a-metadata-standard.aspx
**** QUESTIONS FOR ESRI TECHNICAL SUPPORT
Background: My current configuration is ArcGIS Desktop version 10 with service pack 1 installed, running under Windows 7 operating system (x64). I also have the FGDC metadata Add-in installed. I currently don’t have a computer with version 9.3 installed, but I do have some feature classes created in 9.3 that I was using to import into Version 10.
I read through the blogs you sent earlier, they were very helpful. I also went through most of the help on the online ArcGIS 10 help files for metadata. I think I have a fairly good understanding of the theory behind the new metadata tools and the mechanics of the process. There are a few areas I am still fuzzy on, thus the questions. We are a contractor on a federal project requiring updates twice a year, and are updating metadata each time. The federal client wants it in FGDC CDGSM format, but have not given states or contractors specific guidance or specifications, so we have some flexibility.
Our workflow in the recent update was to import the September, 2010 metadata from 9.3 format into Version 10, modify and add some process steps. I chose to maintain it in ArcGIS metadata format and have the ability to export an xml in FGDC CDGSM format if it is requested in that format. As a result, I chose to do my current updates with the standard ArcGIS metadata editor after upgrading the 9.3 feature classes. Here were the workflow steps I followed:
1) Set the ArcCatalog Options Metadata tab Metadata Style to FGDC CSDGM Metadata
2) Upgrade 9.3 metadata using the Upgrade metadata Tool (using the standalone version of ArcCatalog ArcToolBox), using the "FGDC to ArcGIS" Upgrade type
3) Export the result using the ArcGIS2FGDC.xml Translator
4) Select the target metadata (the version 10 feature class) (for current feature class in ArcCatalog and click import, select "FROM_FGDC" import type
5) Make updates to the metadata for the current submission
My Questions TO ESRI:
1) In this instance I chose not to edit in the FGDC metadata add in, but I did experiment with it on a 9.3 feature class that had not been upgraded. It didn’t seem to change the blue colored FGDC Metadata 9.3 metadata heading at the bottom of the metadata. The help implies that if you edit and change items using the add in, it is stored in the metadata, but can that be viewed anywhere except in the add in editor, or an XML file? Can edits be exported? My take on the help is they can with the USGS MP Metadata Translator tool in the Arc Toolbox, but only from an XML file, not the editor? Once I upgrade the metadata from 9.3 the add in is empty (which I expected), but there were still some remnants left in the blue colored FGDC Metadata 9.3 metadata heading at the bottom of the metadata, can those be edited and deleted?
2) What is the best practice workflow to export FGDC CSDGM metadata from metadata maintained in ArcGIS Metadata format? Is the best way to Export from the top of the editor using the using the ArcGIS2FGDC.xml Translator, then follow up using the USGS MP Metadata Translator on the resulting XML file?
3) Prior to actually exporting, a validation is likely an obvious best practice. I have a few sub-questions relating to that. I got error messages when I ran validate, which I presume came from errors in my original metadata. At this point since the metadata had been upgraded, I assume I either go back to a copy before upgrading and update in the FGDC add in iteratively reimporting, etc. until errors are corrected, OR try and figure out where those items should be placed in the new ArcGIS metadata editor to avoid validation errors. Are there any crosswalk or schematics in the new format to know where things might be placed to be exported in the appropriate output standard, or is it just trial and error? Specifically, I was looking for where the FGDC mandatory fields "Logical Consistency" and "Completeness" might be entered. In the 9.3 FGDC editor they were under the data quality tab and were exported immediately before the process steps. I can’t find those anywhere in the new editor.
4) Is there any feature specific or context specific help with the ArcGIS metadata? For instance, under the Quality item the first choice is Level Scope and there are 25 types, but I couldn't find an help on those or any of the other fields in the editor. Is there a separate white paper or anything on them?
5) There does not appear to be a way to delete a process step once it is created, one can only blank it out, though it then appears as a blank process step. There is also no way to insert of rearrange process steps once they are created, except a lot of manual cutting and pasting that I could find.
**** ANSWERS FROM ESRI TECHNICAL SUPPORT
… concerning your questions about metadata. I have taken ownership of this incident and will be working with you to reach a resolution to this matter.
Regarding your questions and workflow:
In general ArcGIS metadata and FGDC metadata are maintained separately.
1. If you edit with the FGDC add-in you will be editing the FGDC metadata, but if you edit with the edit button you will be editing the ArcGIS metadata that was created from importing the FGDC metadata. Maintaining ArcGIS metadata should be fine if you want to export to FGDC later. If you choose to edit the FGDC metadata using the add in, it should be able to be seen if you set your style sheet as FGDC. The FGDC style sheet is only available by downloading the FGDC style patch at http://resources.arcgis.com/content/patches-and-service-packs?fa=viewPatch&PID=160&MetaID=1637.
2. Is the best way to Export from the top of the editor using the using the ArcGIS2FGDC.xml Translator? Yes, this is the best way (no USGS MP Metadata Translator necessary).
3. Regarding your question about the logical consistency report and completeness sections of the metadata. This information is imported to ArcGIS metadata on upgrade, but you are running into a known issue as these sections are there, but they are not available for editing in the ArcGIS format. The bug is NIM063208 [Unable to edit the Data Quality portion of metadata in the ArcCatalog metadata editor.] This issue is resolved in Service Pack 2 of ArcGIS 10.
4 and 5. I am checking on your question about additional documentation for ArcGIS metadata style and the problem with process steps. I will let you know what I find.
*** Final observations
I'm not positive, but I think I already had the FGDC translator (ArcGIS2FGDC.xml) used for export and the FGDC CSGDM Metadata style on the Metadata tab parameter under ArcCatalog Options before the patch (I assume the latter is what the patch fixes?) Perhaps it is loaded with Service pack 1? Anyway, I downloaded and installed the patch. I confirmed I can edit metadata on a 9.3 file using the FGDC add-in tool (this installs the 9.3 FGDC metadata tool into ArcGIS 10) that had not been upgraded yet. At first I was expecting the edit to take effect after saving. It doesn’t appear until I select another feature class and then come back and open the original one again. DON’T UPGRADE YOUR METADATA IF YOU WANT TO DO THIS.
Option 1. Keep using the 9.3 metadata editor in ArcGIS 10, using the FGDC add-in tool
1) Upgrade to ArcGIS 10, service pack 1
2) Install the FGDC add-in tool
3) Install the patch
4) DO NOT Upgrade 9.3 metadata using the Upgrade Metadata Tool
5) Edit the metadata inside the FGDC add-in tool (outside of the add-in tool in the ESRI metadata description, your edits will appear below the ESRI default metadata (many fields will be blank because you don’t want to upgrade. The full FGDC metadata will appear at the bottom under the blue FGDC Metadata item)
6) Optionally export it to an FGDC XML standalone file using the ArcGIS2FGDC.xml Translator in the second parameter in the export dialog box
Option 2: Upgrade to ArcGIS metadata format and editor and maintain the metadata in the new format, and trust that eventually (service pack 2?) you will be able to export it in truly compliant FGDC format (or one of the other standards)
1) Install ArcGIS 10, service pack 1
2) Install the add-in
3) Install the patch
4) Set the ArcCatalog Options Metadata tab Metadata Style to FGDC CSDGM Metadata
5) Upgrade 9.3 metadata using the Upgrade Metadata Tool (using the standalone version of ArcCatalog ArcToolBox), using the "FGDC to ArcGIS" Upgrade type
6) Make updates to the metadata for the current submission in the ArcGIS metadata editor
Optionally export it to an FGDC XML standalone file using the ArcGIS2FGDC.xml Translator in the second parameter in the export dialog box (you will get most of the FGDC fields, but not all).
