Auto Process Icefield Orientation
- Former user (Deleted)
- jr_roth
Using the Programs
Gather the necessary files
Download the Icefield tool orientation data from the palm pilot and follow the steps in MI5 Multishot Orientation Tool User Guide up to, but not including the section titled Retrieving Standpipe Pressure files and Picking Orientations. From these steps, you will have generated a .prn, .edt, and a .out. Note where these are saved because eventually, all will need to be uploaded to MUT. For now, the Project_orientation program is only interested in the .prn file.
From LORE, download the Core summary file pertaining to the hole you have just oriented and save this to a location you will be able to access. This will be used to determine the upper and lower time limits of each core. For example, if core 2 was one deck at 1300 hrs and core 3 was on deck at 1330 hrs, we know that any data between the time that core 3 was on deck 1330hrs and when core 2 was on deck 1300hrs, belongs to core 3.
Download the ascii standpipe pressure files from rigwatch. A new file is generally generated every 10 minutes. Grab all files from the start of the orientation survey to the end of the survey (or till the last core in the survey is on deck). If you grab more files than necessary (i.e. outside the survey window) it will not pose a problem but the program will run slower. Save these files to a location you will be able to access.
Be sure you note which column the pressure data is in. It should stay in the same column for the duration of the expedition but may change from one expedition to the next. Ask the Ops superintendent if you are unsure.
Note the start time of the survey as it was entered in the orientation notebook in GMT. It is critical that the start of the survey is recorded with the utmost accuracy using GMT time from the ships computers. The reason being is that the program sorts all of the data (standpipe pressures and orientation data) by time and compares these times to the time on deck from the core summary file. If your recorded survey start time is off by say 5 minutes, the pressure files will not align with the orientation data; it will appear offset by 5 minutes. Given that the cores are only oriented for 5 minutes, this could result in the program selecting data entirely outside of the actual orientation window.
This situation is not catastrophic and can be fixed fairly easily when looking at the final xlsx file. It will, however, result in the program not choosing an accurate (or a shortened) orientation period. You will then need to select the orientation period manually.
Note which cores where oriented. Generally, this is straight forward and is just the first core on the selected survey through the last core on the survey. Things can occasional be more complicated. For instance, you may have a survey where one or more cores were not oriented, then orienting was resumed on the same survey. This is important to know since you will need to enter all the cores which were oriented. If you enter a core that was not oriented, the program will search for data which matches the stabilization criteria. If the tool was sitting on the drill rig floor motionless, it will appear to be stable and the program may select this data as an orientation period if you don't use the standpipe pressures.
Running the Program
Open the exe program. This is saved on the downhole core desc. computer at C:\Orientation and has a shortcut on the desktop. The program is also saved on the share drive at T:\IODP_Share\PMag\PMag_Software\orientation tool auto picks and can be run from any windows computer. The program will create a Hole sub directory i.e U9999A in whichever folder you have the .exe saved. Within this subdirectory there will be multiple folders created but the one of most concern to the user is the for_MUT folder. The temp files folder could be useful for troubleshooting or further analysis of data. More on those folders later.
When the program opens, it should pull up an interface such as shown in Figure 1. It will also open a black command prompt screen. This has been enabled for debugging and error tracking purposes and can be disabled in later versions. For now, just minimize the command prompt screen. The program may take a minute or two to load so be patient. The black command prompt will open first.
Figure. 1 GUI Interface
Enter the expedition number, hole identifier, cores oriented, and date and time the survey started. Be sure to use the format specified in the program.
expedition number should be entered as: Exp395E
hole identifier should be entered as: U1560A
Click the button that says locate.prn file. Navigate to where you saved the files from step 1 and select the .prn file.
Do the same for the Locate Core Summary File.
The program will give you a choice to use the standpipe pressure files or not. If you choose not to use the standpipe pressure files, do not click on the Locate Standpipe Pressure File button and leave the field where you select the pressure column as the default (it wont matter if you change it because it will be ignored if you do not load any pressure files).
The program will generally select accurate orientation periods without the standpipe pressure files but there are certain cases where the use of the standpipe pressure files will prevent the program from selecting irrelevant data. For example, if the APC temperature tool is run, it will be held steady for 10 minutes after the core has been fired. By using the standpipe pressure files, the program will only select data that is stable (low magnetic tool face standard deviation AND is before the core barrel was fired within a 6 minute window. This fine tunes the criteria and prevents unwanted data from being included in the average.
If standpipe pressure files are used (recommended), navigate to where you saved the ascii files and select all the files for that survey. Be sure to select which column the pressure data is stored in.
Figure 2
Enter the standard deviation value to be used to identify stable periods. This should stay the same for the duration of a hole and may take some experimentation to see what works best for your particular expedition. The default value is 3 and has been used for all testing to date and has worked well. The actual standard deviation during orienting is much less (closer to 1 or less) 3 still provides accurate results.
Once you have entered all the required fields, hit the big green Run This will start the processing. Depending on the length of the survey, it could take up to 5 minutes to complete. Most of this time is spent sorting through all of the standpipe pressure files. These are large files since there is a data point taken very second. When the processing is complete, a window will pop up saying it was successful and will provide a path to where the output files are saved. Click ‘ok’.
If the program came across any cores that did not satisfy the stability criteria, a warning window will pop up saying which cores did not meet stability criteria and that those cores where not averaged and included in the .csv file that gets uploaded to MUT. In this case, you will need to open the .xlsx file click on the tab for that core and select the data during the orientation period and average it manually, you can then copy and past these cells containing the averaged values into the .csv file for upload to MUT.
Background
What the Program Does
The program reads in the orientation data from the .prn file. This data has a time column but for whatever reason, it is not accurate and the difference in time is variable so it cannot be used. The datapoints are assigned a new time based on the time the user enters. This step is similar to what had been done in the past manually. Next, the standpipe pressure files are read in. The program takes all the files that the user selects, converts them into .csv files which python is better able to work with, then combines them into one file. The program then drops all columns except the time column and the pressure column(which the user can specify).
This next step is one of the most useful features of the program in sorting out the data. At the very least, it should make manually processing the data easier. The core summary file is read into the program and the time on deck for each core is extracted from this file. Using the time on deck, the program sets an upper and lower time limit for each core. Any time in between core X’s time on deck and the previous cores’ time on deck must inherently belong to core X. Based on these upper and lower limits, the data points from the orientation files, and the standpipe pressure files are assigned to a core.
This time period can be long depending on how long it took to retrieve the core etc.. Thus, we are not really interested in all of the data in this time period. We want the period that the core is being oriented. The program determines this period in up to 2 ways. First, it looks for stable values in the magnetic tool face (Tool-M) similar to what a user would do manually. To determine stability, the program calculates a rolling standard deviation over a window of 5 readings (50 seconds). If the standard deviation is less than the user specified standard deviation, the program determines this is stable and this is the period of orientation. 5 readings was chosen because if the period is too short, it may select data that just happens to be stable for a couple readings but isn’t within the true orientation window. If the period is too long, the orientation window will appear artificially shortened by the program. This criteria alone is generally accurate in determining the true orientation window but there are occasions where the tool may be sitting still for some reason but not actually orienting a core. If this happens between the upper and lower time limit of a core, the program will assume this data belongs to the orientation window and will include it. This is not desirable but it can be prevented by including the second (optional) method for selecting the orientation window.
The second method for determining the orientation window is using the standpipe pressure files. Standpipe pressure will increase to close to 3000 psi when a core is fired. Prior to firing the core, the core barrel (with the orientation tool) is left stationary for a period of 5 minutes to orient the core. This is the window we are looking for. Because standpipe pressure can be variable and does not always go up to 3000 psi when the core is fired, the program searches for peaks with a prominence of 2000 psi or more. These peaks are when the core has been fired. Because there may be more than one shot for core 1 (if the mudline was missed) the last peak before the core was on deck is selected as the fire time. The program determines that anything 6 minutes before the core is fired up to when the core is fired is the window of orientation. The data must ALSO satisfy the standard deviation criteria during this window. A window of 6 minutes was chosen versus 5 minutes in order to give a buffer window in case survey start time does not match exactly with rig watch time. Just because data is within the 6 minute window does not mean it is automatically included. It must also be stable (i.e. low standard deviation).
The program takes the data that it determines is within the orientation window and calculates the arithmetic mean of the values. This is done for each core that was oriented. Magnetic tool face and azimuth are handled slightly different since they are angles measured from 0-360 degrees. These measurements have the possibility of being off by 180 degrees if the arithmetic mean is calculated. An example of this would be if magnetic tool face values are clustered around 0°. In order to ensure the mean is correct, each angle is converted to its x and y vector components. All the x components are averaged together as are the y components. The average x and y vector component is then converted back to an angle in degrees.
The average from each column is added to a .csv file formatted for upload to MUT. This file is given the same name as the .prn file which was uploaded. This is to ensure all the file names are the same (with the exception of the extension). Any cores which did not satisfy stabilization criteria or that the program otherwise determined were not oriented are not included in this file. The program gives a warning saying which cores are not included.
A second file is also generated and saved to the same file location. This is the .xlsx file which also gets uploaded to MUT (again with the same file name as the .prn file).T his file contains the raw data from the .prn file with the corrected time (similar to what is done manually). The program creates a separate tab (or worksheet) for each core. This tab contains all the data points from both the standpipe files and the orientation data which belong the that core. The series lengths will be different because the standpipe files record every second while the orientation tool records every 10 seconds. A chart is generated plotting Magnetic Tool Face and Magnetic Tool Face standard deviation vs Time (you will need to expand the chart to make it readable). If standpipe pressure files were uploaded, the 6 minute orientation period based on core firing will be plotted on the same chart as steady green line (series value of 1). This chart provides a quick visual deference for how the data looks and can make it easy to identify if something does not look right. The user can also see which rows of orientation data were selected by the program if they use some conditional formatting to highlight rows which have a std less then the specified value AND contain a value of 1 (or any value) in the oriented column. This column contains a value of 1 during the 6 minute window after the core was fired.
Figure 3 .xlsx output graph
Figure 4 .xlsx output graph showing temperature tool
If the user wishes to override the program selected average, they can select the rows which were oriented and average them with an excel formula. This row of averaged values can be copy and pasted into the .csv file for upload to MUT. Keep in mind that azimuth and Magnetic tool face may be 180 degrees off so take a look at these to make sure they are accurate. The user will be required to do this for cores that did not meet the criteria.
Other Files Generated
A couple of other files will be generated and saved to the temp files folder within the hole directory. These generally should not be needed but may be useful for troubleshooting or sorting out where problems may have originated from. The first of these is an xlsx file with the same name as the files to be uploaded. This file will be identical to the .csv file which gets uploaded to MUT. A .xlsx file is created as an intermediate step in the process. You shouldn’t need this file at all.
Core_times: This is a file that appends the upper and lower limits of each core to the core summary. This is printed just to confirm that the times are assigned correctly. You will notice that only cores that the user enters in the interface are given upper and lower limits. The first core does not have a lower limit assigned in the spread sheet. The lower limit for this core is set at the start of the survey.
Fire_times: This is a spreadsheet that notes the time that each core was fired based on the maximum standpipe pressure achieved between each cores time limits.
For_qc is the right hand dataframe in the .xlsx file for MUT upload. Anything in this file can be vied from the xlsx file so you shouldn’t need this.
Oriented_cores_only: This is a spreadsheet which contains all the datapoints (and only these datapoints) that satisfy the orientation criteria; either Magnetic tool face standard deviation only or magnetic tool face standard deviation AND within the 6 minute window of when the core was fired. These are the values that get averaged. This file could be useful but the user could also so this information (and more) from the xlsx file with some conditional formatting.
Original: This is the raw data with the addition of the core designations
Original_with_standpipe: this is the raw data with the addition of core designations, standard devation, and a column (Oriented)which records a value of ‘1’ if the datapoint is withing the core fired window.
Stand_times_df: This is pretty useless and will remove in the next iteration. It is a list of all the times in the standpipe files. It was used to ensure proper formatting during development but now has no use.
Standpipe: This is the left dataframe in the xlsx file for upload to MUT. It contains the time and pressure from all the standpipe files. Other columns are the core number, max pressure achieved during each core, and the time of that max pressure.
Standpipe_sorted: This is a subset of standpipe which contains one row for each core that was oriented. This row is the highest pressure that was achieved for that core.
Standpipe_w_max: This doesn’t show anything that standpipe doesn’t. It is an intermediate dataframe which contains the max pressure for each column before. Standpipe then adds the time column to this.
There is also a subdirectory called standpipe files. These are files that the program creates as intermediate steps.
Pall: Contains the time and pressure columns from the concatenated standpipe pressure files.
The subdirectory csv created: this contains all of the csvs which are created from the raw ascii standpipe files. These are created so python can read them with more functionality than ascii files which have close to zero functionality in python.
Most of these files are pretty redundant and may not be of any use. The majority do not even need to be generated for the program to function. They were invaluable during development to see what was going on behind the scenes and may prove useful for that during use if problems arise. This is why I have left them in.